When a user interacts with a web application, the application sends multiple requests to a server to fetch data or perform actions. These requests are called network calls, and they contain critical information that can help developers debug and optimize the application. In this post, we will discuss how to use Selenium and Python to extract useful information from network calls such as request headers, request payload, etc.
Manually capture the network calls
We can access the developer tools in modern web browsers to capture and view network calls by right-clicking on the webpage and selecting the “inspect” option.
Clicking on the “Inspect” option will open the developer tools, where there are a bunch of tabs. We can click the “Network” tab to capture the network calls.
Capture network request calls using Selenium and Python
Enable Logging
By default, logging is disabled in Selenium. However, we can enable it using the following code:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
chrome_options.add_experimental_option('detach',True)
is added so that the browser doesn’t close automatically.
Get Performance Logs
After enabling logs, we can retrieve them with the get_log() method. We would need the “performance” logs because they contain network call information.
log_entries = driver.get_log("performance")
We can print the log_entries to see how the output is returned from the get_log() function.
for entry in log_entries:
print(entry)
Below is a part of the output printed in the console –
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.makemytrip.com/","frameId":"C19EDD405671198E42011B294C45B423","hasUserGesture":false,"initiator":{"columnNumber":335,"lineNumber":102,"type":"parser","url":"https://www.makemytrip.com/"},"loaderId":"0556D8BD87F9E5B84B89EFA6E04FF4E6","redirectHasExtraInfo":false,"request":{"headers":{"Referer":"https://www.makemytrip.com/","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\""},"initialPriority":"Medium","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.makemytrip.com/tNpzX7b1O0d352Asb6aBnZQG6No/O7EtQVL4zwEtf5/YkMABnMC/RDReNT/NWVyQ"},"requestId":"30940.17","timestamp":1185069.736025,"type":"Script","wallTime":1706935095.366664}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095373}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.makemytrip.com/","frameId":"C19EDD405671198E42011B294C45B423","hasUserGesture":false,"initiator":{"columnNumber":422,"lineNumber":102,"type":"parser","url":"https://www.makemytrip.com/"},"loaderId":"0556D8BD87F9E5B84B89EFA6E04FF4E6","redirectHasExtraInfo":false,"request":{"headers":{"Referer":"https://www.makemytrip.com/","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\""},"initialPriority":"Medium","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.makemytrip.com/_sec/cp_challenge/sec-4-1.css"},"requestId":"30940.18","timestamp":1185069.736084,"type":"Stylesheet","wallTime":1706935095.366722}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095373}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.makemytrip.com/","frameId":"C19EDD405671198E42011B294C45B423","hasUserGesture":false,"initiator":{"columnNumber":101,"lineNumber":103,"type":"parser","url":"https://www.makemytrip.com/"},"loaderId":"0556D8BD87F9E5B84B89EFA6E04FF4E6","redirectHasExtraInfo":false,"request":{"headers":{"Referer":"https://www.makemytrip.com/","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\""},"initialPriority":"Low","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.makemytrip.com/_sec/cp_challenge/sec-cpt-4-1.js"},"requestId":"30940.19","timestamp":1185069.73613,"type":"Script","wallTime":1706935095.366766}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095373}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSentExtraInfo","params":{"associatedCookies":[{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":-1,"httpOnly":false,"name":"isGdprRegion","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":true,"size":13,"sourcePort":443,"sourceScheme":"Secure","value":"0"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354751,"httpOnly":false,"name":"ccde","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":6,"sourcePort":443,"sourceScheme":"Secure","value":"IN"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.354876,"httpOnly":false,"name":"dvid","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":40,"sourcePort":443,"sourceScheme":"Secure","value":"0030495a-4de0-4d73-a611-58368315b8bc"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354923,"httpOnly":false,"name":"lang","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":7,"sourcePort":443,"sourceScheme":"Secure","value":"eng"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.355087,"httpOnly":false,"name":"_abck","path":"/","priority":"Medium","sameParty":false,"secure":true,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706942294.355143,"httpOnly":false,"name":"ak_bmsc","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":507,"sourcePort":443,"sourceScheme":"Secure","value":"408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706949495.355202,"httpOnly":false,"name":"bm_sz","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128"}}],"clientSecurityState":{"initiatorIPAddressSpace":"Public","initiatorIsSecureContext":true,"privateNetworkRequestPolicy":"PreflightWarn"},"connectTiming":{"requestTime":1185069.75151},"headers":{":authority":"www.makemytrip.com",":method":"GET",":path":"/tNpzX7b1O0d352Asb6aBnZQG6No/O7EtQVL4zwEtf5/YkMABnMC/RDReNT/NWVyQ",":scheme":"https","accept":"*/*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cookie":"isGdprRegion=0; ccde=IN; dvid=0030495a-4de0-4d73-a611-58368315b8bc; lang=eng; _abck=282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1; ak_bmsc=408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7; bm_sz=2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128","referer":"https://www.makemytrip.com/","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\"","sec-fetch-dest":"script","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"},"requestId":"30940.17","siteHasCookieInOtherPartition":false}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095388}
{'level': 'INFO', 'message': '{"message":{"method":"Network.requestWillBeSentExtraInfo","params":{"associatedCookies":[{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":-1,"httpOnly":false,"name":"isGdprRegion","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":true,"size":13,"sourcePort":443,"sourceScheme":"Secure","value":"0"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354751,"httpOnly":false,"name":"ccde","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":6,"sourcePort":443,"sourceScheme":"Secure","value":"IN"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.354876,"httpOnly":false,"name":"dvid","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":40,"sourcePort":443,"sourceScheme":"Secure","value":"0030495a-4de0-4d73-a611-58368315b8bc"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1709527095.354923,"httpOnly":false,"name":"lang","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":7,"sourcePort":443,"sourceScheme":"Secure","value":"eng"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1738471095.355087,"httpOnly":false,"name":"_abck","path":"/","priority":"Medium","sameParty":false,"secure":true,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706942294.355143,"httpOnly":false,"name":"ak_bmsc","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":507,"sourcePort":443,"sourceScheme":"Secure","value":"408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7"}},{"blockedReasons":[],"cookie":{"domain":".makemytrip.com","expires":1706949495.355202,"httpOnly":false,"name":"bm_sz","path":"/","priority":"Medium","sameParty":false,"secure":false,"session":false,"size":434,"sourcePort":443,"sourceScheme":"Secure","value":"2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128"}}],"clientSecurityState":{"initiatorIPAddressSpace":"Public","initiatorIsSecureContext":true,"privateNetworkRequestPolicy":"PreflightWarn"},"connectTiming":{"requestTime":1185069.752303},"headers":{":authority":"www.makemytrip.com",":method":"GET",":path":"/_sec/cp_challenge/sec-4-1.css",":scheme":"https","accept":"text/css,*/*;q=0.1","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cookie":"isGdprRegion=0; ccde=IN; dvid=0030495a-4de0-4d73-a611-58368315b8bc; lang=eng; _abck=282E832A86CEF8A57B18CCF6FCF7AE78~-1~YAAQjV06F6vVvWCNAQAAB5hCbQt5DIx8uYhz8kJqno8XOzYbFoAsehVEFdTfcpxYkLbnBZCbgubLdLzvGF1tAI57kTWqZJQiXNN3vZaUS6WqnSsV5jVjWbfnUl4kodvXBUdkjLf3uZQViRqudsJQmpulZ+dCGKU48sglVrsLDtzmgR3JKWJvLTlNIhsFxMPIl32fQfUDjGwmn/yMVT6jyxKHYzHumhBe4LHVfSFDvOhb+UerYl3R2j8xa40O2jCl12fBUHSLLJ+ex8QByofHxG8BnfMk798/quI3j+h76BA3IBjCTGk1mUX3tWcbMSOEWV4MECMPcZNZ+SXozVByaCftUXIU/iwF0HZlFH314RJLmHT9F8xAAVBmaKiDrA7Y~-1~-1~-1; ak_bmsc=408142DE043E71D1F168EC6B2EF260AB~000000000000000000000000000000~YAAQjV06F6zVvWCNAQAAB5hCbRbpzDC8JHITbefh4PmRVaPoLiHGDCNTKPyaENtkf/n71fin69ERwGcJKxdhARpv+Ij5xs1ncR3443Ku7oN+7gYdRY5ikB3S0SWp7dopxwtKX3TCBlGlSSl6U2ajIjyQG83ePLfqqz5WkrAS434g+1NOK80FgODmpeZ+7ZHfN2jr5fChqdkZAdJRlcatifa662AmV5I/HILc2Qv9wg2uFUlL6/QQgN5qa+iCGbZH6e89MPpcwtJ266qwitvqXBSXdHvfF0/O8Nu0wQxn3T1MZ7Lom0w0QkwwittaBwG5AUqRNg2UIJEWbxmRMIO4A7DcBwOAJB5s8+C3Eii32ouwe5DE3FZp9f/+UhpA+AOVq4iCoM4Zq8Jq8ZUAt/w6swg5imdWv3koMAfnzo9+9ENy4t+Um6u7; bm_sz=2CFAD44A8B20C2EF49C8C8796E0D5CDB~YAAQjV06F67VvWCNAQAAB5hCbRbA8rGCX5aD+H6aw7o5PqadPuTOuAQUp4TjXIgtY34tMnfY8oRV7VDCRxsMFD9vtbWsIUuuHqaZN1/3xV1KLhrTvf0HQPodZ2ofl6wkkj1VVml9ywUrvxncR8S304zhTYPAq0BUHEHogdmC7qNvzKyyfRoCsXaVSLdNCc1JxxTNijhzyZ3KiZDoeSUgdEETifAlqBvWWH9FZTNv7ZFXcw6eyiSP0SWOT5IjP8Ymt7X6FrJSxBReZuUOqmqOxp+V8OE+rzSdSdnwASFI65nWZ7UErpfGca18Ok24LT4lIKkVLOPa1PSCDY2PVYT8hl16JCXTzA2MSXMrQnHT2XWdLh6BSoEIeIZk+SU=~3420471~3491128","referer":"https://www.makemytrip.com/","sec-ch-ua":"\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\\"Windows\\"","sec-fetch-dest":"style","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"},"requestId":"30940.18","siteHasCookieInOtherPartition":false}},"webview":"C19EDD405671198E42011B294C45B423"}', 'timestamp': 1706935095388}
We can parse the above output and fetch the relevant information needed to know the request sent to the server in various network calls.
Capture network call request
The “Network.requestWillBeSent” event captures the requests sent to the server. This event is captured in the log_entries as well, and the corresponding JSON can be parsed to extract request parameters such as headers and payload.
Here is one sample JSON of requestWillBeSent event
{
"method": "Network.requestWillBeSent",
"params": {
"documentURL": "https://www.makemytrip.com/",
"frameId": "C19EDD405671198E42011B294C45B423",
"hasUserGesture": false,
"initiator": {
"stack": {
"callFrames": [
{
"columnNumber": 188345,
"functionName": "",
"lineNumber": 1,
"scriptId": "63",
"url": "https://jsak.mmtcdn.com/pwa_v3/pwa_commons/build/client/js/3548.0d4710c35fb8b8657b66.js"
},
{
"columnNumber": 185659,
"functionName": "xhr",
"lineNumber": 1,
"scriptId": "63",
"url": "https://jsak.mmtcdn.com/pwa_v3/pwa_commons/build/client/js/3548.0d4710c35fb8b8657b66.js"
},
{
"columnNumber": 189264,
"functionName": "_t",
"lineNumber": 1,
"scriptId": "63",
"url": "https://jsak.mmtcdn.com/pwa_v3/pwa_commons/build/client/js/3548.0d4710c35fb8b8657b66.js"
}
]
},
"type": "script"
},
"loaderId": "0556D8BD87F9E5B84B89EFA6E04FF4E6",
"redirectHasExtraInfo": false,
"request": {
"hasPostData": true,
"headers": {
"Accept": "application/json",
"Authorization": "h4nhc9jcgpAGIjp",
"Content-Type": "application/json",
"Referer": "https://www.makemytrip.com/",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"currency": "inr",
"deviceid": "0030495a-4de0-4d73-a611-58368315b8bc",
"language": "eng",
"os": "desktop",
"region": "in",
"sec-ch-ua": "\\"Not_A Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"120\\", \\"Google Chrome\\";v=\\"120\\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\\"Windows\\"",
"tid": "0030495a-4de0-4d73-a611-58368315b8bc",
"usr-mcid": "0030495a-4de0-4d73-a611-58368315b8bc",
"vid": "0030495a-4de0-4d73-a611-58368315b8bc",
"visitor-id": "0030495a-4de0-4d73-a611-58368315b8bc"
},
"initialPriority": "High",
"isSameSite": true,
"method": "POST",
"mixedContentType": "none",
"postData": "{\\"brand\\":\\"MMT\\",\\"profileType\\":\\"B2C\\"}",
"postDataEntries": [
{
"bytes": "eyJicmFuZCI6Ik1NVCIsInByb2ZpbGVUeXBlIjoiQjJDIn0="
}
],
"referrerPolicy": "strict-origin-when-cross-origin",
"url": "https://mapi.makemytrip.com/mobile-core-api-web/api/home/config/v1?brand=mmt&profile=b2c&device=dt&version=NT%2010.0®ion=in&language=eng¤cy=inr"
},
"requestId": "30940.74",
"timestamp": 1185070.570752,
"type": "XHR",
"wallTime": 1706935096.2019
}
},
"webview": "C19EDD405671198E42011B294C45B423"
}
We can simply parse the above JSON to extract the relevant information.
Extracting request URL
request_url - message -> params -> request -> url
Extracting request method
request_url - message -> params -> request -> method
Extracting request headers
request_url - message -> params -> request -> headers
Extracting request payload
request_url - message -> params -> request -> payload
Below is the code to extract the request parameters from log_entries
for entry in log_entries:
try:
message_obj = json.loads(entry.get("message"))
message = message_obj.get("message")
method = message.get("method")
if method == 'Network.requestWillBeSent':
request_url = message.get('params', {}).get('request', {}).get('url', '')
request_method = message.get('params', {}).get('request', {}).get('method', '')
request_headers = message.get('params', {}).get('request', {}).get('headers', {})
request_payload = message.get('params', {}).get('request', {}).get('postData', {})
print("Request URL: "+request_url)
print("Request Method: "+request_method)
print("Request Headers: "+request_headers)
print("Request Payload: "+request_payload)
except Exception as e:
print(e)
Whole code –
from selenium import webdriver
import json
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.makemytrip.com")
# enable network tracking
driver.execute_cdp_cmd('Network.enable', {})
# Capture network log entries
log_entries = driver.get_log("performance")
for entry in log_entries:
try:
message_obj = json.loads(entry.get("message"))
message = message_obj.get("message")
method = message.get("method")
if method == 'Network.requestWillBeSent':
request_url = message.get('params', {}).get('request', {}).get('url', '')
request_method = message.get('params', {}).get('request', {}).get('method', '')
request_headers = message.get('params', {}).get('request', {}).get('headers', {})
request_payload = message.get('params', {}).get('request', {}).get('postData', {})
print("Request URL: "+request_url)
print("Request Method: "+request_method)
print("Request Headers: "+request_headers)
print("Request Payload: "+request_payload)
except Exception as e:
print(e)
This is it. We hope that you have liked the article. If you have any doubts or concerns, please write to us in the comments or mail us at admin@codekru.com.