DSpace Statistics Graham Triggs Head of Repository Systems

DSpace Statistics Graham Triggs Head of Repository Systems, Symplectic

A Brief History

Statistics in DSpace 1. 0

Statistics in DSpace 1. 1 This slide is left intentionally blank

Statistics in DSpace 1. 2 If I’m honest, this is just padding

Statistics in DSpace 1. 3

Classic Statistics Shows items archived, views, searches Parses dspace. log Renders flat HTML files Uses two scripts which must be scheduled Reports can be public, or admin only

Classic Statistics – Config All configuration in [dspace]/config/dstat. cfg (overview and search exclusions) Displays: Overview Archive breakdown (item types) Items viewed Actions (Deletion, Update, Create, etc. ) Logins Searches (keywords) Action names in [dspace]/config/dstat. map

Classic Statistics – Issues dspace. log is primarily for debugging May not log all information required May log lots of unnecessary information Size of log files 1 log line does not equal a single access No filtering of spiders, robots, etc. Log parsing may take some time Slow to update stats

Fast Forward: DSpace 1. 6

Solr Statistics Available for JSP and XML Uis Event logger writes to Apache Solr Filters Spiders by IP address Reports are searches of usage data Reports can be public, or admin only

Solr Stats - What is Indexed Time Type (item, bitstream, etc), Id Owning Community, Owning Collection, Owning Item IP, Continent, Country, City, Longtitude / Latitude Eperson Id, User Agent Flag to indicate Robot / Spider

Solr Stats – Home Top 10 items

Solr Stats – Community Total visits Visits last 7 months Top 10 Countries Top 10 Cities

Solr Stats – Collection Total visits Visits last 7 months Top 10 Countries Top 10 Cities

Solr Stats – Item Total visits Total file views Visits last 7 months Top 10 Countries Top 10 Cities

Solr Stats – Config (v 1. 6) [dspace]/config/dspace. cfg solr. log. server solr. dbfile Filter out spider IP addresses in query solr. query. filter. is. Bot Client identification when hosted behind proxy solr. query. filter. spider. Ip URLs to download IP addresses of search spiders use. Proxies Location of Geo database solr. spiderips. url Location of Solr server / application Filter out ‘is. Bot’ field in query statistics. item. authorization. admin Set to ‘true’ to restrict to admins, false for public access

Solr Stats – Config (v 1. 7) [dspace]/config/dspace. cfg solr. log. server solr. dbfile solr. spiderips. url use. Proxies solr. query. filter. spider. Ip solr. query. filter. is. Bot statistics. item. authorization. admin solr. resolver. timeout Timeout for the DNS resolver (lower for fewer connections) solr. satatistics. log. Bots Disable logging of events by spider IP addresses

Solr Stats – Config (v 1. 8) [dspace]/config/modules/solr-statistics. cfg server spiderips. urls dbfile resolver. timeout use. Proxies log. Bots query. filter. spider. Ip query. filter. is. Bot authorization. admin query. filter. bundles Bundles for which to display file stats (requires 1. 8 index)

Solr Stats – Improvements Dspace v 1. 8 Displayed file bundle Configurable - defaults to ORIGINAL bundle [dspace]/bin/dspace stats-util –b –r Dspace v 1. 7 Solr Optimization [dspace]/bin/stats-util –o Autocommit Defaults to 15 minute intervals Configurable in [dspace]/solr/statistics/colrconfig. xml max. Time property

Solr Stats – Upgrade from Classic Scripts parse dspace. log files to Solr entries [dspace]/bin/dspace stats-log-converter [dspace]/bin/dspace stats-log-importer -I Input file -m Adds a wildcard to the input (i. e. dspace. log*) -s Skip reverse DNS lookup (can be slow) -v Verbose output

Solr Stats –Custom Queries You can expand the reports by querying the Solr index directly Example: Top downloads for a user – query on epersonid facet: <lst name="facet_counts"> <lst name="facet_fields"> <lst name="epersonid"> <int name="66">1167</int> <int name="117">251</int> <int name="52">42</int> <int name="19">36</int> <int name="88">20</int> <int name="112">18</int> <int name="110">9</int> <int name="96">0</int> </lst>

Solr Stats - Maintenance [dspace]/bin/dspace stats-util –h usage: Statistics. Client -b, --reindex-bitstreams Reindex the bitstreams to ensure we have the bundle name -r, --remove-deleted-bitstreams While indexing the bundle names remove the statistics about deleted bitstreams -u, --update-spider-files Update Spider IP Files from internet into /dspace/config/spiders -f, --delete-spiders-by-flag Delete Spiders in Solr By is. Bot Flag -i, --delete-spiders-by-ip Delete Spiders in Solr By IP Address -m, --mark-spiders Update is. Bot Flag in Solr -h, --help help -o, --optimize Run maintenance on the SOLR index

Solr Stats - Issues Privacy laws – IP addresses not anonymized Performance issues / resource usage Maintenance of Solr Usage when Solr is unavailable Usage tracking during periods of high usage

Summary Classic Statistics Possibly slow to analyse, fast to display Delay in updating Very imperfect Solr Statistics Updates ‘real time’ Can be slow to render as dataset grows Improved in each release Less imperfect