🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 102 (from laksa165)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa102.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://dzone.com/articles/how-to-create-question-answering-model-from-scratc\')), getAhrefsUnparsedNoserviceFromURL(\'https://dzone.com/articles/how-to-create-question-answering-model-from-scratc\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc","crawl_time":1775172220,"first_indexed_time":1683754338,"http_code":200,"src_unparsed":"com,dzone!\/articles\/how-to-create-question-answering-model-from-scratc s443","src_root_hash":"8443605588788139902","history_drop_reason":null,"meta_title":"How To Create a Question-Answering Model From Scratch","meta_descriptions":[""],"attrs_boilerpipe_text":"Likes\n(4)\nSave\n6.3K Views\nJoin the DZone community and get the full member experience.\nJoin For Free\nBuilding your question-answering model might seem like it could be more manageable. This tutorial will walk you through step by step to get it done.\nQuestion and Answering is a natural language processing (NLP) technique used to answer questions based on data given or text. Question Answering models aim to develop a system that automatically understands questions passed in natural language and provides accurate or relevant answers. Question answering is widely used in various disciplines, such as Chatbots and virtual assistants, Education and E-Learning, information retrieval, and search engines.\nSetting Development Environment\nBefore continuing to this tutorial, you need to have some knowledge of Python to follow quickly through this setup and development process. Navigate to Google Colab or activate your Jupyter Notebook. Create a new Notebook. Then execute the following commands to install the required libraries in your development environment.\nYou will use transformers to provide pre-trained models and tools for working with state-of-the-art NLP models.\nImporting Necessary Libraries\nImport the required libraries that you will use to preprocess the data and create the model.\nImported libraries will be used later in the code to develop the model.\nDefine Context and List of Questions Based on Context\nDefine a context string that contains the information you want to ask questions about. Here we have chosen a storyline about mountain formation.\nNext, create a list of questions you want to ask about the given context.\nEncode the first question using the tokenizer’s encode method, specifying truncation and padding. Encoding helps improve the model’s consistency, memory efficiency, and training performance.\nCreate a question-answering pipeline using the BERT-based model and tokenizer.\nLoad the Pre-Trained BERT-Based Model and Tokenizer\nBERT (Bidirectional Encoder Representations from Transformers) mostly used language representation models in NLP tasks and questions answering. Loading pre-trained BERT-based models increases the chances of the model understanding language and its capabilities for question answering. Tokenization will divide the input text into meaningful units for the model to process.\nCombining BERT and tokenizer increases your model’s performance and accurate result.\nObtaining Answers\nNow that you have got context and the pre-trained model loaded, you need to pass access to the answers by assigning the answers to a dictionary and printing the score together with the answer. Like in the above, we have two questions and will assign each question by its index. The model will print the answers and the score in each.\nWow! Just simply, you have your question-answering model using Hugging Face Transformers.\nConclusion\nFollowing this tutorial, you have a basic QA model using the\nHugging Face Transformers library\n. By following this tutorial, you can quickly adapt the code to work with different contexts and questions, allowing you to create your question-answering model for various contexts.\nRemember that the example code provided is just a starting point, and you can explore and customize it further based on your specific needs and requirements. Have fun experimenting and building more advanced question-answering models!\nComplete code available on\nGITHUB REPO\nNLP\nQuestion answering\nOpinions expressed by DZone contributors are their own.","attrs_markdown":"[![DZone](https:\/\/dz2cdn1.dzone.com\/themes\/dz20\/images\/dz_logo_2021_cropped.png)](https:\/\/dzone.com\/)\n\nThanks for visiting DZone today,\n\n![user avatar]()\n\n[Edit Profile](https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc)\n\n- [Manage Email Subscriptions](https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc)\n- [How to Post to DZone](https:\/\/dzone.com\/articles\/how-to-submit-a-post-to-dzone?utm_source=DZone&utm_medium=user_dropdown&utm_campaign=how_to_post)\n- [Article Submission Guidelines](https:\/\/dzone.com\/articles\/dzones-article-submission-guidelines)\n\n[Sign Out](https:\/\/dzone.com\/users\/logout.html) [View Profile](https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc)\n\nPost\n\n- ![](https:\/\/dz2cdn1.dzone.com\/themes\/dz20\/images\/dz-postarticle.svg) [Post an Article](https:\/\/dzone.com\/content\/article\/post.html)\n- [Manage My Drafts](https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc)\n\nOver 2 million developers have joined DZone.\n\n[Log In](https:\/\/dzone.com\/users\/login.html) \/ [Join](https:\/\/dzone.com\/static\/registration.html)\n\nSearch\n\nPlease enter at least three characters to search\n\nSearch\n\nPlease enter at least three characters to search\n\n[Refcards](https:\/\/dzone.com\/refcardz) [Trend Reports](https:\/\/dzone.com\/trendreports)\n\n[Events](https:\/\/dzone.com\/events) [Video Library](https:\/\/dzone.com\/events\/video-library)\n\n[Refcards](https:\/\/dzone.com\/refcardz)\n\n[Trend Reports](https:\/\/dzone.com\/trendreports)\n\nEvents\n\n[View Events](https:\/\/dzone.com\/events) [Video Library](https:\/\/dzone.com\/events\/video-library)\n\n### Related\n- [Large Language Models: A Comprehensive Analysis of Real-World CX Applications](https:\/\/dzone.com\/articles\/large-language-models-a-comprehensive-analysis-of-real-world-cx-applications)\n- [Unlocking Hidden Value in Dirty Data: A Practical NLP Pattern for Legacy Records](https:\/\/dzone.com\/articles\/unlocking-hidden-value-in-dirty-data-a-practical-n)\n- [Scaling QA Processes for Enterprise App Development: A Practical Guide](https:\/\/dzone.com\/articles\/Scaling-QA-Processes-for-Enterprise-App-Development)\n- [Building a Customer Intelligence AI Agent With OpenSearch and LLMs](https:\/\/dzone.com\/articles\/build-customer-intelligence-ai-opensearch-llms)\n\n### Trending\n- [Anthropic’s Model Context Protocol (MCP): A Developer’s Guide to Long-Context LLM Integration](https:\/\/dzone.com\/articles\/anthropics-model-context-protocol-mcp-a-developers)\n- [Apache Spark 4.0: What’s New for Data Engineers and ML Developers](https:\/\/dzone.com\/articles\/apache-spark-40-whats-new-for-data-engineers-and-ml-devs)\n- [Essential Techniques for Production Vector Search Systems Part 2 - Binary Quantization](https:\/\/dzone.com\/articles\/production-vector-search-techniques-binary-quantization)\n- [Enriching AI With Real-Time Insights via MCP](https:\/\/dzone.com\/articles\/enriching-ai-with-real-time-insights-via-mcp)\n\n1. [DZone](https:\/\/dzone.com\/)\n2. [Data Engineering](https:\/\/dzone.com\/data-engineering)\n3. [AI\/ML](https:\/\/dzone.com\/ai-ml)\n4. How To Create a Question-Answering Model From Scratch\n\n# How To Create a Question-Answering Model From Scratch\n### Building your question-answering model might seem like it could be more manageable. This tutorial will walk you through step by step to get it done.\nBy\n\n![Joseph owino user avatar](https:\/\/dz2cdn1.dzone.com\/thumbnail?fid=16757810&w=80)\n\n[Joseph owino](https:\/\/dzone.com\/users\/4874090\/joseyusuf.html)\n\n·\n\nMay. 10, 23  · Tutorial\n\nLikes (4)\n\nLikes\n\n![](https:\/\/dzone.com\/articles\/user.profile.profileImage)\n\nThere are no likes...yet! 👀\n\nBe the first to like this post\\!\n\nIt looks like you're not logged in.\n\n[Sign in](https:\/\/dzone.com\/users\/login.html) to see who liked this post\\!\n\nLoad More\n\nComment (0)\n\nSave\n\n[Tweet](https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc)\n\n[Share](https:\/\/www.linkedin.com\/sharing\/share-offsite\/?url=https:\/\/dzone.com\/articles\/how-to-create-question-answering-model-from-scratc)\n\n6\\.3K Views\n\nJoin the DZone community and get the full member experience.\n\n[Join For Free](https:\/\/dzone.com\/static\/registration.html)\n\nBuilding your question-answering model might seem like it could be more manageable. This tutorial will walk you through step by step to get it done.\n\nQuestion and Answering is a natural language processing (NLP) technique used to answer questions based on data given or text. Question Answering models aim to develop a system that automatically understands questions passed in natural language and provides accurate or relevant answers. Question answering is widely used in various disciplines, such as Chatbots and virtual assistants, Education and E-Learning, information retrieval, and search engines.\n\n## **Setting Development Environment**\nBefore continuing to this tutorial, you need to have some knowledge of Python to follow quickly through this setup and development process. Navigate to Google Colab or activate your Jupyter Notebook. Create a new Notebook. Then execute the following commands to install the required libraries in your development environment.\n\nPython\n\n1\n\n1\n```\n!pip install transformers\n```\n\nYou will use transformers to provide pre-trained models and tools for working with state-of-the-art NLP models.\n\n## **Importing Necessary Libraries**\nImport the required libraries that you will use to preprocess the data and create the model.\n\nPython\n\n3\n\n1\n```\nimport tensorflow as tf\n```\n\n2\n```\nfrom transformers import pipeline\n```\n\n3\n```\nfrom transformers import BertForQuestionAnswering\n```\n\nImported libraries will be used later in the code to develop the model.\n\n## Define Context and List of Questions Based on Context\nDefine a context string that contains the information you want to ask questions about. Here we have chosen a storyline about mountain formation.\n\nPython\n\n2\n\n1\n```\nquestion_answerer =pipeline(\"question-answering\")\n```\n\n2\n```\ncontext = \"A mountain is an elevated portion of the Earth's crust, generall\n```\n\nNext, create a list of questions you want to ask about the given context.\n\nPython\n\n4\n\n1\n```\nquestions = [\n```\n\n2\n```\n\"What is a mountain?\",\n```\n\n3\n```\n\"What is a plateau?\"\n```\n\n4\n```\n]\n```\n\nEncode the first question using the tokenizer’s encode method, specifying truncation and padding. Encoding helps improve the model’s consistency, memory efficiency, and training performance.\n\nPython\n\n1\n```\ntokenizer.encode(questions[0], truncation=True, padding=True)\n```\n\nCreate a question-answering pipeline using the BERT-based model and tokenizer.\n\nPython\n\n6\n\n1\n```\nquestion_answerer=pipeline('question-answering', model=model, tokenizer=tok\n```\n\n2\n```\nquestion_answerer({\n```\n\n3\n```\n'question':questions,\n```\n\n4\n```\n'context' : context;\n```\n\n5\n```\n})\n```\n\n6\n```\n\n```\n## **Load the Pre-Trained BERT-Based Model and Tokenizer**\nBERT (Bidirectional Encoder Representations from Transformers) mostly used language representation models in NLP tasks and questions answering. Loading pre-trained BERT-based models increases the chances of the model understanding language and its capabilities for question answering. Tokenization will divide the input text into meaningful units for the model to process.\n\nCombining BERT and tokenizer increases your model’s performance and accurate result.\n\nPython\n\n3\n\n1\n```\nfrom transformers import AutoTokenizer\n```\n\n2\n```\nmodel = BertForQuestionAnswering.from_pretrained(\"deepset\/bert-base-cased-s\n```\n\n3\n```\ntokenizer = AutoTokenizer.from_pretrained(\"deepset\/bert-base-cased-squad2\")\n```\n## **Obtaining Answers**\nNow that you have got context and the pre-trained model loaded, you need to pass access to the answers by assigning the answers to a dictionary and printing the score together with the answer. Like in the above, we have two questions and will assign each question by its index. The model will print the answers and the score in each.\n\nPython\n\nx\n\n1\n```\nquestion_answerer({\n```\n\n2\n```\n'question': questions[0],\n```\n\n3\n```\n'context': context\n```\n\n4\n```\n})\n```\n\n5\n```\nquestion_answerer({\n```\n\n6\n```\n'question': questions[1],\n```\n\n7\n```\n'context': context\n```\n\n8\n```\n})\n```\n\n9\n```\n\n```\n\nWow! Just simply, you have your question-answering model using Hugging Face Transformers.\n\n## Conclusion\nFollowing this tutorial, you have a basic QA model using the [Hugging Face Transformers library](https:\/\/dzone.com\/articles\/getting-started-with-hugging-face-transformers-for). By following this tutorial, you can quickly adapt the code to work with different contexts and questions, allowing you to create your question-answering model for various contexts.\n\nRemember that the example code provided is just a starting point, and you can explore and customize it further based on your specific needs and requirements. Have fun experimenting and building more advanced question-answering models\\!\n\nComplete code available on [GITHUB REPO](https:\/\/github.com\/josephotienoowino\/Question-and-answering\/blob\/main\/q%26a_with_huggingface.py)\n\nNLP Question answering\n\nOpinions expressed by DZone contributors are their own.\n\n### Related\n- Large Language Models: A Comprehensive Analysis of Real-World CX Applications\n- Unlocking Hidden Value in Dirty Data: A Practical NLP Pattern for Legacy Records\n- Scaling QA Processes for Enterprise App Development: A Practical Guide\n- Building a Customer Intelligence AI Agent With OpenSearch and LLMs\n## Partner Resources\n\n×\n\n***\nComments\n\nOops! Something Went Wrong\n\nThe likes didn't load as expected. Please refresh the page and try again.\n\nClose\n\nABOUT US\n\n- [About DZone](https:\/\/dzone.com\/pages\/about)\n- [Support and feedback](mailto:support@dzone.com)\n- [Community research](https:\/\/dzone.com\/pages\/dzone-community-research)\n\nADVERTISE\n\n- [Advertise with DZone](https:\/\/advertise.dzone.com\/)\n\nCONTRIBUTE ON DZONE\n\n- [Article Submission Guidelines](https:\/\/dzone.com\/articles\/dzones-article-submission-guidelines)\n- [Become a Contributor](https:\/\/dzone.com\/pages\/contribute)\n- [Core Program](https:\/\/dzone.com\/pages\/core)\n- [Visit the Writers' Zone](https:\/\/dzone.com\/writers-zone)\n\nLEGAL\n\n- [Terms of Service](https:\/\/technologyadvice.com\/terms-conditions\/)\n- [Privacy Policy](https:\/\/technologyadvice.com\/privacy-policy\/)\n\nCONTACT US\n\n- 3343 Perimeter Hill Drive\n- Suite 215\n- Nashville, TN 37211\n- [support@dzone.com](mailto:support@dzone.com)\n\nLet's be friends:\n\nLikes\n\n![](https:\/\/dzone.com\/articles\/user.profile.profileImage)\n\nThere are no likes...yet! 👀\n\nBe the first to like this post\\!\n\nIt looks like you're not logged in.\n\n[Sign in](https:\/\/dzone.com\/users\/login.html) to see who liked this post\\!\n\nLoad More","attrs_readable_markdown":null,"meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

7 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.2 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://dzone.com/articles/how-to-create-question-answering-model-from-scratc
Last Crawled	2026-04-02 23:23:40 (7 days ago)
First Indexed	2023-05-10 21:32:18 (2 years ago)
HTTP Status Code	200
Meta Title	How To Create a Question-Answering Model From Scratch
Meta Description	null
Meta Canonical	null
Boilerpipe Text	Likes (4) Save 6.3K Views Join the DZone community and get the full member experience. Join For Free Building your question-answering model might seem like it could be more manageable. This tutorial will walk you through step by step to get it done. Question and Answering is a natural language processing (NLP) technique used to answer questions based on data given or text. Question Answering models aim to develop a system that automatically understands questions passed in natural language and provides accurate or relevant answers. Question answering is widely used in various disciplines, such as Chatbots and virtual assistants, Education and E-Learning, information retrieval, and search engines. Setting Development Environment Before continuing to this tutorial, you need to have some knowledge of Python to follow quickly through this setup and development process. Navigate to Google Colab or activate your Jupyter Notebook. Create a new Notebook. Then execute the following commands to install the required libraries in your development environment. You will use transformers to provide pre-trained models and tools for working with state-of-the-art NLP models. Importing Necessary Libraries Import the required libraries that you will use to preprocess the data and create the model. Imported libraries will be used later in the code to develop the model. Define Context and List of Questions Based on Context Define a context string that contains the information you want to ask questions about. Here we have chosen a storyline about mountain formation. Next, create a list of questions you want to ask about the given context. Encode the first question using the tokenizer’s encode method, specifying truncation and padding. Encoding helps improve the model’s consistency, memory efficiency, and training performance. Create a question-answering pipeline using the BERT-based model and tokenizer. Load the Pre-Trained BERT-Based Model and Tokenizer BERT (Bidirectional Encoder Representations from Transformers) mostly used language representation models in NLP tasks and questions answering. Loading pre-trained BERT-based models increases the chances of the model understanding language and its capabilities for question answering. Tokenization will divide the input text into meaningful units for the model to process. Combining BERT and tokenizer increases your model’s performance and accurate result. Obtaining Answers Now that you have got context and the pre-trained model loaded, you need to pass access to the answers by assigning the answers to a dictionary and printing the score together with the answer. Like in the above, we have two questions and will assign each question by its index. The model will print the answers and the score in each. Wow! Just simply, you have your question-answering model using Hugging Face Transformers. Conclusion Following this tutorial, you have a basic QA model using the Hugging Face Transformers library . By following this tutorial, you can quickly adapt the code to work with different contexts and questions, allowing you to create your question-answering model for various contexts. Remember that the example code provided is just a starting point, and you can explore and customize it further based on your specific needs and requirements. Have fun experimenting and building more advanced question-answering models! Complete code available on GITHUB REPO NLP Question answering Opinions expressed by DZone contributors are their own.
Markdown	[![DZone](https://dz2cdn1.dzone.com/themes/dz20/images/dz_logo_2021_cropped.png)](https://dzone.com/) Thanks for visiting DZone today, ![user avatar]() [Edit Profile](https://dzone.com/articles/how-to-create-question-answering-model-from-scratc) - [Manage Email Subscriptions](https://dzone.com/articles/how-to-create-question-answering-model-from-scratc) - [How to Post to DZone](https://dzone.com/articles/how-to-submit-a-post-to-dzone?utm_source=DZone&utm_medium=user_dropdown&utm_campaign=how_to_post) - [Article Submission Guidelines](https://dzone.com/articles/dzones-article-submission-guidelines) [Sign Out](https://dzone.com/users/logout.html) [View Profile](https://dzone.com/articles/how-to-create-question-answering-model-from-scratc) Post - ![](https://dz2cdn1.dzone.com/themes/dz20/images/dz-postarticle.svg) [Post an Article](https://dzone.com/content/article/post.html) - [Manage My Drafts](https://dzone.com/articles/how-to-create-question-answering-model-from-scratc) Over 2 million developers have joined DZone. [Log In](https://dzone.com/users/login.html) / [Join](https://dzone.com/static/registration.html) Search Please enter at least three characters to search Search Please enter at least three characters to search [Refcards](https://dzone.com/refcardz) [Trend Reports](https://dzone.com/trendreports) [Events](https://dzone.com/events) [Video Library](https://dzone.com/events/video-library) [Refcards](https://dzone.com/refcardz) [Trend Reports](https://dzone.com/trendreports) Events [View Events](https://dzone.com/events) [Video Library](https://dzone.com/events/video-library) ### Related - [Large Language Models: A Comprehensive Analysis of Real-World CX Applications](https://dzone.com/articles/large-language-models-a-comprehensive-analysis-of-real-world-cx-applications) - [Unlocking Hidden Value in Dirty Data: A Practical NLP Pattern for Legacy Records](https://dzone.com/articles/unlocking-hidden-value-in-dirty-data-a-practical-n) - [Scaling QA Processes for Enterprise App Development: A Practical Guide](https://dzone.com/articles/Scaling-QA-Processes-for-Enterprise-App-Development) - [Building a Customer Intelligence AI Agent With OpenSearch and LLMs](https://dzone.com/articles/build-customer-intelligence-ai-opensearch-llms) ### Trending - [Anthropic’s Model Context Protocol (MCP): A Developer’s Guide to Long-Context LLM Integration](https://dzone.com/articles/anthropics-model-context-protocol-mcp-a-developers) - [Apache Spark 4.0: What’s New for Data Engineers and ML Developers](https://dzone.com/articles/apache-spark-40-whats-new-for-data-engineers-and-ml-devs) - [Essential Techniques for Production Vector Search Systems Part 2 - Binary Quantization](https://dzone.com/articles/production-vector-search-techniques-binary-quantization) - [Enriching AI With Real-Time Insights via MCP](https://dzone.com/articles/enriching-ai-with-real-time-insights-via-mcp) 1. [DZone](https://dzone.com/) 2. [Data Engineering](https://dzone.com/data-engineering) 3. [AI/ML](https://dzone.com/ai-ml) 4. How To Create a Question-Answering Model From Scratch # How To Create a Question-Answering Model From Scratch ### Building your question-answering model might seem like it could be more manageable. This tutorial will walk you through step by step to get it done. By ![Joseph owino user avatar](https://dz2cdn1.dzone.com/thumbnail?fid=16757810&w=80) [Joseph owino](https://dzone.com/users/4874090/joseyusuf.html) · May. 10, 23 · Tutorial Likes (4) Likes ![](https://dzone.com/articles/user.profile.profileImage) There are no likes...yet! 👀 Be the first to like this post\! It looks like you're not logged in. [Sign in](https://dzone.com/users/login.html) to see who liked this post\! Load More Comment (0) Save [Tweet](https://dzone.com/articles/how-to-create-question-answering-model-from-scratc) [Share](https://www.linkedin.com/sharing/share-offsite/?url=https://dzone.com/articles/how-to-create-question-answering-model-from-scratc) 6\.3K Views Join the DZone community and get the full member experience. [Join For Free](https://dzone.com/static/registration.html) Building your question-answering model might seem like it could be more manageable. This tutorial will walk you through step by step to get it done. Question and Answering is a natural language processing (NLP) technique used to answer questions based on data given or text. Question Answering models aim to develop a system that automatically understands questions passed in natural language and provides accurate or relevant answers. Question answering is widely used in various disciplines, such as Chatbots and virtual assistants, Education and E-Learning, information retrieval, and search engines. ## Setting Development Environment Before continuing to this tutorial, you need to have some knowledge of Python to follow quickly through this setup and development process. Navigate to Google Colab or activate your Jupyter Notebook. Create a new Notebook. Then execute the following commands to install the required libraries in your development environment. Python 1 1 ``` !pip install transformers ``` You will use transformers to provide pre-trained models and tools for working with state-of-the-art NLP models. ## Importing Necessary Libraries Import the required libraries that you will use to preprocess the data and create the model. Python 3 1 ``` import tensorflow as tf ``` 2 ``` from transformers import pipeline ``` 3 ``` from transformers import BertForQuestionAnswering ``` Imported libraries will be used later in the code to develop the model. ## Define Context and List of Questions Based on Context Define a context string that contains the information you want to ask questions about. Here we have chosen a storyline about mountain formation. Python 2 1 ``` question_answerer =pipeline("question-answering") ``` 2 ``` context = "A mountain is an elevated portion of the Earth's crust, generall ``` Next, create a list of questions you want to ask about the given context. Python 4 1 ``` questions = [ ``` 2 ``` "What is a mountain?", ``` 3 ``` "What is a plateau?" ``` 4 ``` ] ``` Encode the first question using the tokenizer’s encode method, specifying truncation and padding. Encoding helps improve the model’s consistency, memory efficiency, and training performance. Python 1 ``` tokenizer.encode(questions[0], truncation=True, padding=True) ``` Create a question-answering pipeline using the BERT-based model and tokenizer. Python 6 1 ``` question_answerer=pipeline('question-answering', model=model, tokenizer=tok ``` 2 ``` question_answerer({ ``` 3 ``` 'question':questions, ``` 4 ``` 'context' : context; ``` 5 ``` }) ``` 6 ``` ``` ## Load the Pre-Trained BERT-Based Model and Tokenizer BERT (Bidirectional Encoder Representations from Transformers) mostly used language representation models in NLP tasks and questions answering. Loading pre-trained BERT-based models increases the chances of the model understanding language and its capabilities for question answering. Tokenization will divide the input text into meaningful units for the model to process. Combining BERT and tokenizer increases your model’s performance and accurate result. Python 3 1 ``` from transformers import AutoTokenizer ``` 2 ``` model = BertForQuestionAnswering.from_pretrained("deepset/bert-base-cased-s ``` 3 ``` tokenizer = AutoTokenizer.from_pretrained("deepset/bert-base-cased-squad2") ``` ## Obtaining Answers Now that you have got context and the pre-trained model loaded, you need to pass access to the answers by assigning the answers to a dictionary and printing the score together with the answer. Like in the above, we have two questions and will assign each question by its index. The model will print the answers and the score in each. Python x 1 ``` question_answerer({ ``` 2 ``` 'question': questions[0], ``` 3 ``` 'context': context ``` 4 ``` }) ``` 5 ``` question_answerer({ ``` 6 ``` 'question': questions[1], ``` 7 ``` 'context': context ``` 8 ``` }) ``` 9 ``` ``` Wow! Just simply, you have your question-answering model using Hugging Face Transformers. ## Conclusion Following this tutorial, you have a basic QA model using the [Hugging Face Transformers library](https://dzone.com/articles/getting-started-with-hugging-face-transformers-for). By following this tutorial, you can quickly adapt the code to work with different contexts and questions, allowing you to create your question-answering model for various contexts. Remember that the example code provided is just a starting point, and you can explore and customize it further based on your specific needs and requirements. Have fun experimenting and building more advanced question-answering models\! Complete code available on [GITHUB REPO](https://github.com/josephotienoowino/Question-and-answering/blob/main/q%26a_with_huggingface.py) NLP Question answering Opinions expressed by DZone contributors are their own. ### Related - Large Language Models: A Comprehensive Analysis of Real-World CX Applications - Unlocking Hidden Value in Dirty Data: A Practical NLP Pattern for Legacy Records - Scaling QA Processes for Enterprise App Development: A Practical Guide - Building a Customer Intelligence AI Agent With OpenSearch and LLMs ## Partner Resources × *** Comments Oops! Something Went Wrong The likes didn't load as expected. Please refresh the page and try again. Close ABOUT US - [About DZone](https://dzone.com/pages/about) - [Support and feedback](mailto:support@dzone.com) - [Community research](https://dzone.com/pages/dzone-community-research) ADVERTISE - [Advertise with DZone](https://advertise.dzone.com/) CONTRIBUTE ON DZONE - [Article Submission Guidelines](https://dzone.com/articles/dzones-article-submission-guidelines) - [Become a Contributor](https://dzone.com/pages/contribute) - [Core Program](https://dzone.com/pages/core) - [Visit the Writers' Zone](https://dzone.com/writers-zone) LEGAL - [Terms of Service](https://technologyadvice.com/terms-conditions/) - [Privacy Policy](https://technologyadvice.com/privacy-policy/) CONTACT US - 3343 Perimeter Hill Drive - Suite 215 - Nashville, TN 37211 - [support@dzone.com](mailto:support@dzone.com) Let's be friends: Likes ![](https://dzone.com/articles/user.profile.profileImage) There are no likes...yet! 👀 Be the first to like this post\! It looks like you're not logged in. [Sign in](https://dzone.com/users/login.html) to see who liked this post\! Load More
Readable Markdown	null
Shard	102 (laksa)
Root Hash	8443605588788139902
Unparsed URL	com,dzone!/articles/how-to-create-question-answering-model-from-scratc s443