{"id":152,"date":"2013-07-11T17:27:03","date_gmt":"2013-07-11T15:27:03","guid":{"rendered":"http:\/\/data-science.ch\/wordpress\/?p=152"},"modified":"2014-06-20T14:33:18","modified_gmt":"2014-06-20T12:33:18","slug":"recall-oriented-expert-search-using-relevance-feedback","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/","title":{"rendered":"Recall-Oriented Expert Search Using Relevance Feedback"},"content":{"rendered":"

The information engineering group at InIT has recently successfully concluded a CTI funded project. The project is called \u201cexpert-match\u201d and has been conducted in cooperation with expert group ag<\/a>, a professional recruitment and consulting business head-quartered in Zurich, Switzerland.<\/p>\n

Expert group\u2019s recruiting focus consists of staffing high-profile expert positions (e.g. specialized senior software engineers, senior management, IT architects). Usually, there are at most a handful of people qualified for a recruiting mandate within the relevant geographical region they operate in. The problem is further compounded by the fact that qualifications are rarely obvious, requiring insight into candidates\u2019 former positions and overall skill sets. Exact matching of job descriptions with candidate profiles, such as in a database environment, is thus unlikely to find the desired experts. By implementing an information retrieval (IR) application, we were able to mitigate the problem. The application fully supports iterative searches, especially focussing on relevance feedback.<\/p>\n

\"expert_match_process\"<\/a><\/p>\n

<\/p>\n

Implementation<\/h2>\n

The application is built on existing components, so as to be able to concentrate on the actual business logic and associated interesting research issues. Terrier<\/a> is used as the basic IR matching system. It proved to be stable and flexible, yet not sufficiently scalable for massive applications. For the given use case and order of magnitude of the data, however, it is perfectly suitable. For the graphical user interface, we intended to provide a highly interactive web application and therefore decided to go with GWT<\/a>.<\/p>\n

Approach<\/h2>\n

Relevance feedback plays a vital role in our approach, since it has been shown to increase retrieval effectiveness significantly in many environments, especially recall, i.e. the proportion of total relevant information which is actually found, which is crucial to the business case. Users are provided with feedback facilities in the form of positive and negative document bins. Dragging candidates to either bin influences subsequent search iterations: The query in the next iteration is expanded using selected terms from positively marked documents.<\/p>\n

The relevance feedback mechanisms are built on top of Terrier components. To allow for the full range of positive and negative feedback, some minor extensions were necessary, since Terrier only supports positive feedback document selection out of the box. When querying, documents (resp. their IDs) marked as positive or negative by users are passed to the relevance feedback components. During query processing, the given feedback documents are scrutinized and their most characteristic terms extracted. The terms are added to the query (up to a threshold count) and the entire query is re-weighted using the well-understood Rocchio algorithm [1]<\/a>. Another matching pass is then performed with the resulting modified query, yielding the final ranked list.<\/p>\n

To appreciate the effect of the implemented feedback, consider documents and queries in vector space. Both queries and documents are represented as sparse vectors, where each dimension is the weight of a term drawn from the vocabulary of the entire document collection. The Rocchio algorithm effectively moves the query vector towards an \u201cideal\u201d result document (i.e. the related documents\u2019 centroid) based on the original query and terms from feedback documents with their associated weights. Note that this only holds true if positive feedback is valued higher than negative one. For an explanation of the effect, see the practical\u00a0[2]<\/a> Rocchio algorithm formula below:<\/p>\n

\"rocchio_formula\"
Rocchio formula<\/figcaption><\/figure>\n

These are the main constituents of the formula:<\/span><\/p>\n

\"rocchio_formula_constituents\"<\/a>
Rocchio formula constituents<\/figcaption><\/figure>\n

Rocchio was chosen over probabilistic alternatives for its ease of parameterization. One of the project\u2019s expressed goals was to evaluate optimal configuration values for the application. After evaluation and testing, the contribution parameters were eventually set to \u03b1 = 0.3, \u03b2 = 0.75\u00a0and \u03b3 = 0.25. In other use cases, \u03b1\u00a0is usually left to be 1 as a neutral factor. But as first tests have shown the original queries\u2019 terms to be largely dominant, their contribution was consequently lowered. Feedback becomes largely ineffective if the original query dominates. In case of entering entire job descriptions as queries, which is supported by design, feedback had entirely no visible effect at all. Conversely, skewing parameters towards feedback dominance obsoletes the original query and leads to results which have little relevance to the original information need as entered. Both extremes are undesirable for users. The intention of a formulated user information need, i.e. the query, must be respected and feedback must have visible and beneficial effects, which the previously shown parameter setting achieved for expert-match.<\/span><\/p>\n

Results<\/h2>\n

The expert-match system is successfully used in expert group\u2019s daily business since the first feature-complete version of the application. In fact, the new system strongly affected the business process of searching for experts: instead of only traditional database subtractive filtering, HR consultants now perform an additive IR search in parallel.\u00a0 This enables them to find candidates which would otherwise have been filtered away due to missing data or differing synonymous descriptions. Also, the relevance feedback facilities allow for searches of candidates which are similar to \u201cprototype\u201d employees as requested by their clients, which is a typical business case (e.g. \u201cGet us another Heidi M\u00fcller, she\u2019s a great addition to the team.\u201d).<\/p>\n

Much of the success of the project is due to the consistent implementation of relevance feedback in all parts of the application. It was possible to implement it in such a way because users were ready to provide explicit feedback, since they already spent significant amounts of time on searching.<\/p>\n

\n
\n

<\/a>[1] Rocchio, J. J. (1971). Relevance feedback in information retrieval. In: Salton, G. (1971). The SMART retrieval system\u2014experiments in automatic document processing. pp. 313 – 323<\/p>\n

<\/a>[2] As adopted in the SMART retrieval system<\/p>\n<\/div>\n

Schlagw\u00f6rter: Information Retrieval<\/a>
<\/div>","protected":false},"excerpt":{"rendered":"

The information engineering group at InIT has recently successfully concluded a CTI funded project. The project is called \u201cexpert-match\u201d and has been conducted in cooperation with expert group ag, a professional recruitment and consulting business head-quartered in Zurich, Switzerland. Expert group\u2019s recruiting focus consists of staffing high-profile expert positions (e.g. specialized senior software engineers, senior […]<\/p>\n","protected":false},"author":38,"featured_media":694,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[7,9],"tags":[8],"features":[],"class_list":["post-152","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-research","tag-information-retrieval"],"yoast_head":"\nRecall-Oriented Expert Search Using Relevance Feedback - Data Science made in Switzerland<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Recall-Oriented Expert Search Using Relevance Feedback\" \/>\n<meta property=\"og:description\" content=\"The information engineering group at InIT has recently successfully concluded a CTI funded project. The project is called \u201cexpert-match\u201d and has been conducted in cooperation with expert group ag, a professional recruitment and consulting business head-quartered in Zurich, Switzerland. Expert group\u2019s recruiting focus consists of staffing high-profile expert positions (e.g. specialized senior software engineers, senior […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\" \/>\n<meta property=\"og:site_name\" content=\"Data Science made in Switzerland\" \/>\n<meta property=\"article:published_time\" content=\"2013-07-11T15:27:03+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-06-20T12:33:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1227\" \/>\n\t<meta property=\"og:image:height\" content=\"664\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"riet\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"riet\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\"},\"author\":{\"name\":\"riet\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/#\/schema\/person\/7a43ec283f97a0deb335b4bb971813a8\"},\"headline\":\"Recall-Oriented Expert Search Using Relevance Feedback\",\"datePublished\":\"2013-07-11T15:27:03+00:00\",\"dateModified\":\"2014-06-20T12:33:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\"},\"wordCount\":930,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png\",\"keywords\":[\"Information Retrieval\"],\"articleSection\":[\"Blog\",\"Research\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\",\"url\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\",\"name\":\"Recall-Oriented Expert Search Using Relevance Feedback - Data Science made in Switzerland\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png\",\"datePublished\":\"2013-07-11T15:27:03+00:00\",\"dateModified\":\"2014-06-20T12:33:18+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/#\/schema\/person\/7a43ec283f97a0deb335b4bb971813a8\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage\",\"url\":\"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png\",\"contentUrl\":\"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png\",\"width\":1227,\"height\":664},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/datascience\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Recall-Oriented Expert Search Using Relevance Feedback\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/datascience\/\",\"name\":\"Data Science made in Switzerland\",\"description\":\"Ein Blog der ZHAW Z\u00fcrcher Hochschule f\u00fcr Angewandte Wissenschaften\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/datascience\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/datascience\/#\/schema\/person\/7a43ec283f97a0deb335b4bb971813a8\",\"name\":\"riet\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/69d9cf8f23251316e74cdba474b272383366343f687a9c82f9752fc3c65b4b33?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/69d9cf8f23251316e74cdba474b272383366343f687a9c82f9752fc3c65b4b33?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/69d9cf8f23251316e74cdba474b272383366343f687a9c82f9752fc3c65b4b33?s=96&d=mm&r=g\",\"caption\":\"riet\"},\"url\":\"https:\/\/blog.zhaw.ch\/datascience\/author\/riet\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Recall-Oriented Expert Search Using Relevance Feedback - Data Science made in Switzerland","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/","og_locale":"en_US","og_type":"article","og_title":"Recall-Oriented Expert Search Using Relevance Feedback","og_description":"The information engineering group at InIT has recently successfully concluded a CTI funded project. The project is called \u201cexpert-match\u201d and has been conducted in cooperation with expert group ag, a professional recruitment and consulting business head-quartered in Zurich, Switzerland. Expert group\u2019s recruiting focus consists of staffing high-profile expert positions (e.g. specialized senior software engineers, senior […]","og_url":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/","og_site_name":"Data Science made in Switzerland","article_published_time":"2013-07-11T15:27:03+00:00","article_modified_time":"2014-06-20T12:33:18+00:00","og_image":[{"width":1227,"height":664,"url":"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png","type":"image\/png"}],"author":"riet","twitter_card":"summary_large_image","twitter_misc":{"Written by":"riet","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/"},"author":{"name":"riet","@id":"https:\/\/blog.zhaw.ch\/datascience\/#\/schema\/person\/7a43ec283f97a0deb335b4bb971813a8"},"headline":"Recall-Oriented Expert Search Using Relevance Feedback","datePublished":"2013-07-11T15:27:03+00:00","dateModified":"2014-06-20T12:33:18+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/"},"wordCount":930,"commentCount":0,"image":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png","keywords":["Information Retrieval"],"articleSection":["Blog","Research"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/","url":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/","name":"Recall-Oriented Expert Search Using Relevance Feedback - Data Science made in Switzerland","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage"},"image":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png","datePublished":"2013-07-11T15:27:03+00:00","dateModified":"2014-06-20T12:33:18+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/#\/schema\/person\/7a43ec283f97a0deb335b4bb971813a8"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#primaryimage","url":"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png","contentUrl":"https:\/\/blog.zhaw.ch\/datascience\/files\/2013\/07\/expert_match_process.png","width":1227,"height":664},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/datascience\/recall-oriented-expert-search-using-relevance-feedback\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/datascience\/"},{"@type":"ListItem","position":2,"name":"Recall-Oriented Expert Search Using Relevance Feedback"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/datascience\/#website","url":"https:\/\/blog.zhaw.ch\/datascience\/","name":"Data Science made in Switzerland","description":"Ein Blog der ZHAW Z\u00fcrcher Hochschule f\u00fcr Angewandte Wissenschaften","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/datascience\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/datascience\/#\/schema\/person\/7a43ec283f97a0deb335b4bb971813a8","name":"riet","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/69d9cf8f23251316e74cdba474b272383366343f687a9c82f9752fc3c65b4b33?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/69d9cf8f23251316e74cdba474b272383366343f687a9c82f9752fc3c65b4b33?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/69d9cf8f23251316e74cdba474b272383366343f687a9c82f9752fc3c65b4b33?s=96&d=mm&r=g","caption":"riet"},"url":"https:\/\/blog.zhaw.ch\/datascience\/author\/riet\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/posts\/152","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/users\/38"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/comments?post=152"}],"version-history":[{"count":4,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/posts\/152\/revisions"}],"predecessor-version":[{"id":714,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/posts\/152\/revisions\/714"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/media\/694"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/media?parent=152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/categories?post=152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/tags?post=152"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/datascience\/wp-json\/wp\/v2\/features?post=152"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}