Get network call requests in Selenium Python

selenium-python

When a user interacts with a web application, the application sends multiple requests to a server to fetch data or perform actions. These requests are called network calls, and they contain critical information that can help developers debug and optimize the application. In this post, we will discuss how to use Selenium and Python to extract useful information from network calls such as request headers, request payload, etc.

Manually capture the network calls

We can access the developer tools in modern web browsers to capture and view network calls by right-clicking on the webpage and selecting the “inspect” option.

Click on "Inspect" option

Clicking on the “Inspect” option will open the developer tools, where there are a bunch of tabs. We can click the “Network” tab to capture the network calls.

Click on "Network" tab
Capture network request calls using Selenium and Python
Enable Logging

By default, logging is disabled in Selenium. However, we can enable it using the following code:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})

chrome_options.add_experimental_option('detach',True) is added so that the browser doesn’t close automatically.

Get Performance Logs

After enabling logs, we can retrieve them with the get_log() method. We would need the “performance” logs because they contain network call information.

log_entries = driver.get_log("performance")

We can print the log_entries to see how the output is returned from the get_log() function.

for entry in log_entries:
    print(entry)

Below is a part of the output printed in the console –

{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.makemytrip.com/","frameId":"C19EDD405671198E42011B294C45B423","hasUserGesture":false,"initiator":{"columnNumber":335,"lineNumber":102,"type":"parser","url":"https://www.makemytrip.com/"},"loaderId":"0556D8BD87F9E5B84B89EFA6E04FF4E6","redirectHasExtraInfo":false,"request":{"headers":{"Referer":"https://www.makemytrip.com/","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\""},"initialPriority":"Medium","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.makemytrip.com/tNpzX7b1O0d352Asb6aBnZQG6No/O7EtQVL4zwEtf5/YkMABnMC/RDReNT/NWVyQ"},"requestId":"30940.17","timestamp":1185069.736025,"type":"Script","wallTime":1706935095.366664}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095373}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.makemytrip.com/","frameId":"C19EDD405671198E42011B294C45B423","hasUserGesture":false,"initiator":{"columnNumber":422,"lineNumber":102,"type":"parser","url":"https://www.makemytrip.com/"},"loaderId":"0556D8BD87F9E5B84B89EFA6E04FF4E6","redirectHasExtraInfo":false,"request":{"headers":{"Referer":"https://www.makemytrip.com/","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\""},"initialPriority":"Medium","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.makemytrip.com/_sec/cp_challenge/sec-4-1.css"},"requestId":"30940.18","timestamp":1185069.736084,"type":"Stylesheet","wallTime":1706935095.366722}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095373}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.makemytrip.com/","frameId":"C19EDD405671198E42011B294C45B423","hasUserGesture":false,"initiator":{"columnNumber":101,"lineNumber":103,"type":"parser","url":"https://www.makemytrip.com/"},"loaderId":"0556D8BD87F9E5B84B89EFA6E04FF4E6","redirectHasExtraInfo":false,"request":{"headers":{"Referer":"https://www.makemytrip.com/","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\""},"initialPriority":"Low","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.makemytrip.com/_sec/cp_challenge/sec-cpt-4-1.js"},"requestId":"30940.19","timestamp":1185069.73613,"type":"Script","wallTime":1706935095.366766}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095373}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSentExtraInfo","params":{"associatedCookies":[{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":-1,"httpOnly":false,"name":"isGdprRegion","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":true,"size":13,"sourcePort":443,"sourceScheme":"Secure","value":"0"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354751,"httpOnly":false,"name":"ccde","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":6,"sourcePort":443,"sourceScheme":"Secure","value":"IN"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.354876,"httpOnly":false,"name":"dvid","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":40,"sourcePort":443,"sourceScheme":"Secure","value":"0030495a-4de0-4d73-a611-58368315b8bc"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354923,"httpOnly":false,"name":"lang","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":7,"sourcePort":443,"sourceScheme":"Secure","value":"eng"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.355087,"httpOnly":false,"name":"_abck","path":"/","priority":"Medium","sameParty":false,"secure":true,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706942294.355143,"httpOnly":false,"name":"ak_bmsc","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":507,"sourcePort":443,"sourceScheme":"Secure","value":"408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706949495.355202,"httpOnly":false,"name":"bm_sz","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128"}}],"clientSecurityState":{"initiatorIPAddressSpace":"Public","initiatorIsSecureContext":true,"privateNetworkRequestPolicy":"PreflightWarn"},"connectTiming":{"requestTime":1185069.75151},"headers":{":authority":"www.makemytrip.com",":method":"GET",":path":"/tNpzX7b1O0d352Asb6aBnZQG6No/O7EtQVL4zwEtf5/YkMABnMC/RDReNT/NWVyQ",":scheme":"https","accept":"*/*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cookie":"isGdprRegion=0; ccde=IN; dvid=0030495a-4de0-4d73-a611-58368315b8bc; lang=eng; _abck=282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1; ak_bmsc=408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7; bm_sz=2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128","referer":"https://www.makemytrip.com/","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\"","sec-fetch-dest":"script","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"},"requestId":"30940.17","siteHasCookieInOtherPartition":false}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095388}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSentExtraInfo","params":{"associatedCookies":[{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":-1,"httpOnly":false,"name":"isGdprRegion","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":true,"size":13,"sourcePort":443,"sourceScheme":"Secure","value":"0"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354751,"httpOnly":false,"name":"ccde","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":6,"sourcePort":443,"sourceScheme":"Secure","value":"IN"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.354876,"httpOnly":false,"name":"dvid","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":40,"sourcePort":443,"sourceScheme":"Secure","value":"0030495a-4de0-4d73-a611-58368315b8bc"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354923,"httpOnly":false,"name":"lang","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":7,"sourcePort":443,"sourceScheme":"Secure","value":"eng"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.355087,"httpOnly":false,"name":"_abck","path":"/","priority":"Medium","sameParty":false,"secure":true,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706942294.355143,"httpOnly":false,"name":"ak_bmsc","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":507,"sourcePort":443,"sourceScheme":"Secure","value":"408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706949495.355202,"httpOnly":false,"name":"bm_sz","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128"}}],"clientSecurityState":{"initiatorIPAddressSpace":"Public","initiatorIsSecureContext":true,"privateNetworkRequestPolicy":"PreflightWarn"},"connectTiming":{"requestTime":1185069.752303},"headers":{":authority":"www.makemytrip.com",":method":"GET",":path":"/_sec/cp_challenge/sec-4-1.css",":scheme":"https","accept":"text/css,*/*;q=0.1","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cookie":"isGdprRegion=0; ccde=IN; dvid=0030495a-4de0-4d73-a611-58368315b8bc; lang=eng; _abck=282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1; ak_bmsc=408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7; bm_sz=2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128","referer":"https://www.makemytrip.com/","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\"","sec-fetch-dest":"style","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"},"requestId":"30940.18","siteHasCookieInOtherPartition":false}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095388}

We can parse the above output and fetch the relevant information needed to know the request sent to the server in various network calls.

Capture network call request

The “Network.requestWillBeSent” event captures the requests sent to the server. This event is captured in the log_entries as well, and the corresponding JSON can be parsed to extract request parameters such as headers and payload.

Here is one sample JSON of requestWillBeSent event

{
    "method": "Network.requestWillBeSent",
    "params": {
        "documentURL": "https://www.makemytrip.com/",
        "frameId": "C19EDD405671198E42011B294C45B423",
        "hasUserGesture": false,
        "initiator": {
            "stack": {
                "callFrames": [
                    {
                        "columnNumber": 188345,
                        "functionName": "",
                        "lineNumber": 1,
                        "scriptId": "63",
                        "url": "https://jsak.mmtcdn.com/pwa_v3/pwa_commons/build/client/js/3548.0d4710c35fb8b8657b66.js"
                    },
                    {
                        "columnNumber": 185659,
                        "functionName": "xhr",
                        "lineNumber": 1,
                        "scriptId": "63",
                        "url": "https://jsak.mmtcdn.com/pwa_v3/pwa_commons/build/client/js/3548.0d4710c35fb8b8657b66.js"
                    },
                    {
                        "columnNumber": 189264,
                        "functionName": "_t",
                        "lineNumber": 1,
                        "scriptId": "63",
                        "url": "https://jsak.mmtcdn.com/pwa_v3/pwa_commons/build/client/js/3548.0d4710c35fb8b8657b66.js"
                    }
                ]
            },
            "type": "script"
        },
        "loaderId": "0556D8BD87F9E5B84B89EFA6E04FF4E6",
        "redirectHasExtraInfo": false,
        "request": {
            "hasPostData": true,
            "headers": {
                "Accept": "application/json",
                "Authorization": "h4nhc9jcgpAGIjp",
                "Content-Type": "application/json",
                "Referer": "https://www.makemytrip.com/",
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
                "currency": "inr",
                "deviceid": "0030495a-4de0-4d73-a611-58368315b8bc",
                "language": "eng",
                "os": "desktop",
                "region": "in",
                "sec-ch-ua": "\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"",
                "sec-ch-ua-mobile": "?0",
                "sec-ch-ua-platform": "\\"Windows\\"",
                "tid": "0030495a-4de0-4d73-a611-58368315b8bc",
                "usr-mcid": "0030495a-4de0-4d73-a611-58368315b8bc",
                "vid": "0030495a-4de0-4d73-a611-58368315b8bc",
                "visitor-id": "0030495a-4de0-4d73-a611-58368315b8bc"
            },
            "initialPriority": "High",
            "isSameSite": true,
            "method": "POST",
            "mixedContentType": "none",
            "postData": "{\\"brand\\":\\"MMT\\",\\"profileType\\":\\"B2C\\"}",
            "postDataEntries": [
                {
                    "bytes": "eyJicmFuZCI6Ik1NVCIsInByb2ZpbGVUeXBlIjoiQjJDIn0="
                }
            ],
            "referrerPolicy": "strict-origin-when-cross-origin",
            "url": "https://mapi.makemytrip.com/mobile-core-api-web/api/home/config/v1?brand=mmt&profile=b2c&device=dt&version=NT%2010.0®ion=in&language=eng¤cy=inr"
        },
        "requestId": "30940.74",
        "timestamp": 1185070.570752,
        "type": "XHR",
        "wallTime": 1706935096.2019
    }
},
"webview": "C19EDD405671198E42011B294C45B423"
}

We can simply parse the above JSON to extract the relevant information.

Extracting request URL 

request_url - message -> params -> request -> url

Extracting request method

request_url - message -> params -> request -> method

Extracting request headers

request_url - message -> params -> request -> headers

Extracting request payload

request_url - message -> params -> request -> payload

Below is the code to extract the request parameters from log_entries 

for entry in log_entries:
    try:
        message_obj = json.loads(entry.get("message"))
        message = message_obj.get("message")
        method = message.get("method")

        if method == 'Network.requestWillBeSent':
            request_url = message.get('params', {}).get('request', {}).get('url', '')
            request_method = message.get('params', {}).get('request', {}).get('method', '')
            request_headers = message.get('params', {}).get('request', {}).get('headers', {})
            request_payload = message.get('params', {}).get('request', {}).get('postData', {})

            print("Request URL: "+request_url)
            print("Request Method: "+request_method)
            print("Request Headers: "+request_headers)
            print("Request Payload: "+request_payload)

    except Exception as e:
        print(e)

Whole code –

from selenium import webdriver
import json

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.makemytrip.com")

# enable network tracking
driver.execute_cdp_cmd('Network.enable', {})

# Capture network log entries
log_entries = driver.get_log("performance")

for entry in log_entries:
    try:
        message_obj = json.loads(entry.get("message"))
        message = message_obj.get("message")
        method = message.get("method")

        if method == 'Network.requestWillBeSent':
            request_url = message.get('params', {}).get('request', {}).get('url', '')
            request_method = message.get('params', {}).get('request', {}).get('method', '')
            request_headers = message.get('params', {}).get('request', {}).get('headers', {})
            request_payload = message.get('params', {}).get('request', {}).get('postData', {})

            print("Request URL: "+request_url)
            print("Request Method: "+request_method)
            print("Request Headers: "+request_headers)
            print("Request Payload: "+request_payload)

    except Exception as e:
        print(e)

This is it. We hope that you have liked the article. If you have any doubts or concerns, please write to us in the comments or mail us at admin@codekru.com.

Related Articles –

Get network call responses in Selenium Python

Liked the article? Share this on

Leave a Comment

Your email address will not be published. Required fields are marked *