Crawl interfaces for Norch running inside your browser

Posted earlier on Search Nuggets.

Got an idea a while back on how we could use the JavaScript/Nodejs Search Engine Norch so that the users would have their own search server inside the browser. The main takeaway from this would be that you don’t need to install anything to test the search engine. Since last time, I’ve made a quick logo for Norch, and drawn some more user interfaces. The mockups are mainly about setting up the crawler, which in Norch terms is called Norch Fetch.

Initial Crawl-window

To crawl most pages elegantly and easily, you need five information elements:

  1. Somewhere to start. Which place do you want your crawler to start. You don’t have to specify the domain, we pick the domain name  from the page you’re visiting.
  2. Which links to follow. This is not necessarily the pages you want to crawl. Typically these pages have lists of pages you want to crawl.
  3. Which links not to follow. To not make the crawler go wild, you set some boundaries. Often a page has several URLs.
  4. Which links to crawl. These are the actual pages you’re looking for.
  5. Which links not to crawl.

A simple illustration on the above rules. Norch Fetch doesn’t have all these features yet, but they’re suggested as enhancements.

Selecting which rule type to add

To ensure you’re adding valid rules, it’s a good ting to test first.

Start URL added

The minimum amount of rules needed to start the crawler

Next tasks will be to make a clickable prototype in HTML/CSS and read up on HTML5 local storage/web storage.

All comments on the idea are welcome! 

One thought on “Crawl interfaces for Norch running inside your browser

Disagree, have a comment or want to pitch in? Youre thoughts are more than welcome =)

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s