Please follow below mentioned steps to install SOLR:

(1) Download the latest version from the following link.

http://lucene.apache.org/solr/

( I used Apache Solr 4.3.0 which was latest at that time.)

 

(2) Extract the bundle to your desired location.

 

(3) Open command terminal, go to the folder where you have extracted Solr bundle then go to example folder and enter following command:

 

java -jar start.jar

 

• After above command solr will be started, to access admin go to following url.

http://localhost:8983/solr/#/collection1

 

 

• Indexing xml Documents to Solr

 

(1) Open new command terminal and go to exampledocs folder inside solr-4.3.0/example.

(2) To index any xml (e.g. hd.xml) document to Solr use the following command.

 

java -jar post.jar hd.xml

 

<add>

<doc>

 <field name="id">Mov-1</field>

 <field name="name">The Shawshank Redemption</field>

  <field name="ratings">9.2</field>

 <field name="description">Story about two prisoners who bond and have friendship...</field>

</doc>

</add>

I have created this file in entertainment folder located in solr-4.3.0. To index this file execute following command:

java -jar exampledocs/post.jar entertainment/top10.xml

• Indexing PDF and DOC Files

(1) In order to index PDF or DOC files, first you need to have curl installed, if not installed, then install it by below command:

• For Ubuntu,

sudo apt-get install curl

• Windows can download curl from following link,

http://curl.haxx.se/download.html

 

• set environment variable to use curl.

(2) Now time to edit Solr instance's solrconfig.xml (solr-4.3.0/example/solr/collection1/conf ) where need to add following configuration:

 

(3) Now, create new folder named "extract" anywhere on your system (I have created that folder in a same directory where Solr is installed), and place apache-solr-cell-4.0.0.jar file under that folder from dist directory (you can find it in the Solr distribution archive).

 

After that you need to copy all library files from contrib/extraction/lib/ directory to the "extract" folder which you have created before.

 

(4) In addition to that, we need add following entries in solrconfig.xml file:

 

<lib dir="../../extract" regex=".*\.jar" />

 

(5) To index PDF file, use following command:

 

curl "http://localhost:8983/solr/update/extract?literal.id=fileId&commit=true" -F "[email protected]"

 

Replace <field> value in URL with id which you want to give as file unique id for a file to be indexed, you can fire this command from the folder containing actual file (e.g. fileName.pdf).

Here I have it in a same folder for this example.

Make sure port number and url is same as Solr is running.

 

After firing above command you will be able to see the following in response:

<?xml version="1.0" encoding="UTF-8"?>

<response>

<lst name="responseHeader">

<int name="status">0</int>

<int name="QTime">578</int>

</lst>

</response>

 

To index DOC file repalce fileName.pdf to desired fileName.docx. 

 

 

• Searching File Metadata

Go to  http://localhost:8983/solr/select?q=Shawshank and replace Shawshank by keyword you want to search. You will see the file contains the keyword.

 

 

• Deleting Indexed File

 

To delete a particular indexed file, fire below command from exampledocs folder:

 

java -Ddata=args -jar post.jar "<delete><query>id:fieId</query></delete>"

 

Here <field> would be the same file unique id which we have used while first time indexed the file.

 

If you don’t know the id of file you can view it by visiting http://localhost:8983/solr/select?q=keyword

 

It will show all the files containing searched keyword. Now look for file you want to delete and look for the line:

 

<str name="id">file_id</str>

 

Copy this id and replace this with <fileId> in above delete query.

 

After executing the command, you will get the following response:

 

SimplePostTool version 1.5

POSTing args to http://localhost:8983/solr/update..

COMMITting Solr index changes to http://localhost:8983/solr/update..

Time spent: 0:00:00.261

 

 

• Updating link of Indexed pdf/doc and make it available for download

You can see the file in search results but you will not able to download the file, if you try to download you will get 404 error telling resource not found.

 

How to resolve this error?

Answer is solr does not upload documents on its own server, for this we need to have another web server like tomcat which is running parallal to solr and will support document download.

 

First makes sure your tomcat server is running as mine here:

localhost:8080

 

Now we need to setup actual file URL and its path on tomcat server.

 

Use following command for setting up actual file URL:

 

curl 'localhost:8983/solr/update?commit=true' -H 'Content-type:application/json' -d '[{"id":"fileId","url":{"set":"http://localhost:8080/path-to-file/fileName.pdf(doc)"}}]'

While executing command, replace <fileId> with file unique id and you are ready to go for file downloading in search result.

 

• Making Apache Solr Search Site

 

In order to make Apache Solr search site content, we need to index whole site.

To index whole site, we need web crawler Apache Nutch by which we can index site data. Apache Nutch is easily configurable with Apache Solr.

(1) Download a binary package (apache-nutch-1.x-bin.zip) from here.

(2) Unzip on local drive.

(3) Go to extracted apache-nutch-1.x folder's bin folder (e.g.: apache-nutch-1.x/bin)

(4) Run "./nutch" (Make sure JAVA_HOME variable is set properly).

(5) You can confirm a correct installation if you seeing the following:

Usage: nutch COMMAND

where COMMAND is one of:

 

 crawl             one-step crawler for intranets (DEPRECATED - USE CRAWL SCRIPT INSTEAD)

 readdb            read / dump crawl db

 mergedb           merge crawldb-s, with optional filtering

 readlinkdb        read / dump link db

 inject            inject new urls into the database

 generate          generate new segments to fetch from crawl db

 freegen           generate new segments to fetch from text files

 fetch             fetch a segment's pages

 parse             parse a segment's pages

 readseg           read / dump segment data

 mergesegs         merge several segments, with optional filtering and slicing

 updatedb          update crawl db from segments after fetching

 invertlinks       create a linkdb from parsed segments

 mergelinkdb       merge linkdb-s, with optional filtering

 solrindex         run the solr indexer on parsed segments and linkdb

 solrdedup         remove duplicates from solr

 solrclean         remove HTTP 301 and 404 documents from solr

 parsechecker      check the parser for a given url

 indexchecker      check the indexing filters for a given url

 domainstats       calculate domain statistics from crawldb

 webgraph          generate a web graph from existing segments

 linkrank          run a link analysis program on the generated web graph

 scoreupdater      updates the crawldb with linkrank scores

 nodedumper        dumps the web graph's node scores

 plugin            load a plugin and run one of its classes main()

 junit             runs the given JUnit test

or

 CLASSNAME         run the class named CLASSNAME

 

Most commands print help when invoked w/o parameters.

 

(5) Now edit a file conf/nutch-site.xml with following content:

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>http.agent.name</name>

<value>My Nutch Spider</value>

</property>

</configuration>

 

(6) Create new directory named urls in apache-nutch-1.x root folder

 

(7) Create a file named seeds.txt inside urls directory with the following content (one URL per line for each site you want Nutch to crawl):

 

http://surekhatech.com/

http://nextsiteurl/

 

 

(8) Now edit a file conf/regex-urlfilter.txt and replace as below:

 

# accept anything else
+.

 

with a regular expression matching the domain you wish to crawl.

For example, if you wish to limit the crawl to the surekhatech.com domain, the line should be like below:

 

+^http://([a-z0-9]*\.)*surekhatech.com/

 

This will include any URL in the domain surekhatech.com.

 

(9) Now we are ready to initiate a crawl, use the following parameters:

 

• -dir dir names the directory to put the crawl in.

• -threads threads determines the number of threads that will fetch in parallel.

• -depth depth indicates the link depth from the root page that should be crawled.

• -topN N determines the maximum number of pages that will be retrieved at each level up to the depth.

• Run the following command:

 

bin/nutch crawl urls -dir crawl -depth 3 -topN 5

 

(10) Integrate Solr with Nutch

 

Finally we have both Nutch and Solr installed and setup correctly, Nutch have already created crawl data from the seed URL(s).

Below are some steps to delegate searching to Solr for links to be searchable:

 

• We need to copy schema.xml (schema-solr4.xml for sorl 4.3.0) file from Nutch to Solr as below:

cp ${NUTCH_RUNTIME_HOME}/conf/schema.xml ${APACHE_SOLR_HOME}/example/solr/collection1/conf

• Restart Solr with the command “java -jar start.jar” under ${APACHE_SOLR_HOME}/example

• Run the Solr Index command:

?\bin/nutch solrindex http://127.0.0.1:8983/solr/ crawl/crawldb -linkdb crawl/linkdbCrawl/segments/*

 

If you want to see the raw HTML indexed by Solr, change the content field definition in schema.xml to:

 

<field name="content" type="text" stored="true" indexed="true"/>

(11) Now open http://localhost:8983/solr/collection1/browse and search for any content, you will get search result from site’s contents.

For professional paid support, you may contact us at [email protected] .