Please follow below mentioned steps to install SOLR:
(1) Download the latest version from the following link.
http://lucene.apache.org/solr/
( I used Apache Solr 4.3.0 which was latest at that time.)
(2) Extract the bundle to your desired location.
(3) Open command terminal, go to the folder where you have extracted Solr bundle then go to example folder and enter following command:
java -jar start.jar
- After above command solr will be started, to access admin go to following url.
http://localhost:8983/solr/#/collection1
- Indexing xml Documents to Solr
(1) Open new command terminal and go to exampledocs folder inside solr-4.3.0/example.
(2) To index any xml (e.g. hd.xml) document to Solr use the following command.
java -jar post.jar hd.xml
Now the file hd.xml is indexed and available for search, you can test it using following url:
http://localhost:8983/solr/collection1/browse
Now type any keyword of the file and you would be able to get search results accordingly.
(3) To index all xml documents located in exampledocs folder type the following command:
java -jar post.jar *.xml
(4) To index you custom xml file create xml file as follows ( I have created file top10.xml which contents top 10 movies)
<add>
<doc>
<field name="id">Mov-1</field>
<field name="name">The Shawshank Redemption</field>
<field name="ratings">9.2</field>
<field name="description">Story about two prisoners who bond and have friendship...</field>
</doc>
</add>
I have created this file in entertainment folder located in solr-4.3.0. To index this file execute following command:
java -jar exampledocs/post.jar entertainment/top10.xml
You may find there is error field named ratings in a file. To resolve this, we need to update schema.xml file located in solr-4.3.0/example/solr/collection1/conf directory.
Add the field you want (which is not present), for my case I need to add following fields,
<field name="ratings" type="float" indexed="true" stored="true" multiValued="true"/>
Now restart the server and index the file by:
java -jar exampledocs/post.jar entertainment/top10.xml
Yur file will now be available.
- Indexing PDF and DOC Files
(1) In order to index PDF or DOC files, first you need to have curl installed, if not installed, then install it by below command:
sudo apt-get install curl
- Windows can download curl from following link,
http://curl.haxx.se/download.html
- set environment variable to use curl.
(2) Now time to edit Solr instance's solrconfig.xml (solr-4.3.0/example/solr/collection1/conf ) where need to add following configuration:
<requestHandler name="/update/extract" class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="fmap.content">text</str>
<str name="lowernames">true</str>
<str name="uprefix">attr_</str>
<str name="captureAttr">true</str>
</lst>
</requestHandler>
(3) Now, create new folder named "extract" anywhere on your system (I have created that folder in a same directory where Solr is installed), and place apache-solr-cell-4.0.0.jar file under that folder from dist directory (you can find it in the Solr distribution archive).
After that you need to copy all library files from contrib/extraction/lib/ directory to the "extract" folder which you have created before.
(4) In addition to that, we need add following entries in solrconfig.xml file:
<lib dir="../../extract" regex=".*\.jar" />
(5) To index PDF file, use following command:
curl "http://localhost:8983/solr/update/extract?literal.id=fileId&commit=true" -F "[email protected]"
Replace <field> value in URL with id which you want to give as file unique id for a file to be indexed, you can fire this command from the folder containing actual file (e.g. fileName.pdf).
Here I have it in a same folder for this example.
Make sure port number and url is same as Solr is running.
After firing above command you will be able to see the following in response:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">578</int>
</lst>
</response>
To index DOC file repalce fileName.pdf to desired fileName.docx.
Go to http://localhost:8983/solr/select?q=Shawshank and replace Shawshank by keyword you want to search. You will see the file contains the keyword.
To delete a particular indexed file, fire below command from exampledocs folder:
java -Ddata=args -jar post.jar "<delete><query>id:fieId</query></delete>"
Here <field> would be the same file unique id which we have used while first time indexed the file.
If you don’t know the id of file you can view it by visiting http://localhost:8983/solr/select?q=keyword
It will show all the files containing searched keyword. Now look for file you want to delete and look for the line:
<str name="id">file_id</str>
Copy this id and replace this with <fileId> in above delete query.
After executing the command, you will get the following response:
SimplePostTool version 1.5
POSTing args to http://localhost:8983/solr/update..
COMMITting Solr index changes to http://localhost:8983/solr/update..
Time spent: 0:00:00.261
- Updating link of Indexed pdf/doc and make it available for download
You can see the file in search results but you will not able to download the file, if you try to download you will get 404 error telling resource not found.
How to resolve this error?
Answer is solr does not upload documents on its own server, for this we need to have another web server like tomcat which is running parallal to solr and will support document download.
First makes sure your tomcat server is running as mine here:
localhost:8080
Now we need to setup actual file URL and its path on tomcat server.
Use following command for setting up actual file URL:
curl 'localhost:8983/solr/update?commit=true' -H 'Content-type:application/json' -d '[{"id":"fileId","url":{"set":"http://localhost:8080/path-to-file/fileName.pdf(doc)"}}]'
While executing command, replace <fileId> with file unique id and you are ready to go for file downloading in search result.
- Making Apache Solr Search Site
In order to make Apache Solr search site content, we need to index whole site.
To index whole site, we need web crawler Apache Nutch by which we can index site data. Apache Nutch is easily configurable with Apache Solr.
Steps to Install Apache Nutch:
(1) Download a binary package (apache-nutch-1.x-bin.zip) from here.
(2) Unzip on local drive.
(3) Go to extracted apache-nutch-1.x folder's bin folder (e.g.: apache-nutch-1.x/bin)
(4) Run "./nutch" (Make sure JAVA_HOME variable is set properly).
(5) You can confirm a correct installation if you seeing the following:
Usage: nutch COMMAND
where COMMAND is one of:
crawl one-step crawler for intranets (DEPRECATED - USE CRAWL SCRIPT INSTEAD)
readdb read / dump crawl db
mergedb merge crawldb-s, with optional filtering
readlinkdb read / dump link db
inject inject new urls into the database
generate generate new segments to fetch from crawl db
freegen generate new segments to fetch from text files
fetch fetch a segment's pages
parse parse a segment's pages
readseg read / dump segment data
mergesegs merge several segments, with optional filtering and slicing
updatedb update crawl db from segments after fetching
invertlinks create a linkdb from parsed segments
mergelinkdb merge linkdb-s, with optional filtering
solrindex run the solr indexer on parsed segments and linkdb
solrdedup remove duplicates from solr
solrclean remove HTTP 301 and 404 documents from solr
parsechecker check the parser for a given url
indexchecker check the indexing filters for a given url
domainstats calculate domain statistics from crawldb
webgraph generate a web graph from existing segments
linkrank run a link analysis program on the generated web graph
scoreupdater updates the crawldb with linkrank scores
nodedumper dumps the web graph's node scores
plugin load a plugin and run one of its classes main()
junit runs the given JUnit test
or
CLASSNAME run the class named CLASSNAME
Most commands print help when invoked w/o parameters.
(5) Now edit a file conf/nutch-site.xml with following content:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>My Nutch Spider</value>
</property>
</configuration>
(6) Create new directory named urls in apache-nutch-1.x root folder
(7) Create a file named seeds.txt inside urls directory with the following content (one URL per line for each site you want Nutch to crawl):
http://surekhatech.com/
http://nextsiteurl/
(8) Now edit a file conf/regex-urlfilter.txt and replace as below:
# accept anything else
+.
with a regular expression matching the domain you wish to crawl.
For example, if you wish to limit the crawl to the surekhatech.com domain, the line should be like below:
+^http://([a-z0-9]*\.)*surekhatech.com/
This will include any URL in the domain surekhatech.com.
(9) Now we are ready to initiate a crawl, use the following parameters:
-
-dir dir names the directory to put the crawl in.
-
-threads threads determines the number of threads that will fetch in parallel.
-
-depth depth indicates the link depth from the root page that should be crawled.
-
-topN N determines the maximum number of pages that will be retrieved at each level up to the depth.
-
Run the following command:
bin/nutch crawl urls -dir crawl -depth 3 -topN 5
(10) Integrate Solr with Nutch
Finally we have both Nutch and Solr installed and setup correctly, Nutch have already created crawl data from the seed URL(s).
Below are some steps to delegate searching to Solr for links to be searchable:
cp ${NUTCH_RUNTIME_HOME}/conf/schema.xml ${APACHE_SOLR_HOME}/example/solr/collection1/conf
?\bin/nutch solrindex http://127.0.0.1:8983/solr/ crawl/crawldb -linkdb crawl/linkdbCrawl/segments/*
If you want to see the raw HTML indexed by Solr, change the content field definition in schema.xml to:
<field name="content" type="text" stored="true" indexed="true"/>
(11) Now open http://localhost:8983/solr/collection1/browse and search for any content, you will get search result from site’s contents.
For professional paid support, you may contact us at [email protected] .