Network calls are integral to web applications, enabling communication between a user’s browser and a web server. This communication facilitates the exchange of data that powers dynamic and interactive online experiences. These calls involve requests and responses, where the browser sends a request to the server to retrieve data, and the server responds with the necessary information. In this post, we will discuss how to use Selenium and Python to retrieve the responses of those calls.
Manually capture the network calls
We can access the developer tools in modern web browsers to capture and view network calls by right-clicking on the webpage and selecting the “inspect” option.
Clicking on the “Inspect” option will open the developer tools, where there are a bunch of tabs. We can click the “Network” tab to capture the network calls.
Capture network calls and their responses using Selenium and Python
Enable Logging
By default, logging is disabled in Selenium. However, we can enable it using the following code:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
chrome_options.add_experimental_option('detach',True) is only added so that the browser doesn’t automatically close.
Get Performance Logs
After enabling the logs, we can retrieve various types of logs( browser, client, performance, etc. ) using the get_log() method. For our case, we would need to extract the “Performance” logs, which Include logs related to the browser’s performance, such as network requests, JavaScript execution times, and other performance metrics.
log_entries = driver.get_log("performance")
We can print the log_entries to see how the output is returned from the get_log() function.
for entry in log_entries:
print(entry)
Below is a part of the output that will be printed in the console.
{'level': 'INFO', 'message': '{"message":{"method":"Network.responseReceived","params":{"frameId":"9C41AC99D954B28E130D2324C3EA66CE","hasExtraInfo":true,"loaderId":"86836D729FDC3171EA9C611C9DA5888C","requestId":"21916.5","response":{"alternateProtocolUsage":"unspecifiedReason","connectionId":147,"connectionReused":false,"encodedDataLength":3079,"fromDiskCache":false,"fromPrefetchCache":false,"fromServiceWorker":false,"headers":{"access-control-allow-credentials":"false","access-control-allow-headers":"*","access-control-allow-methods":"GET,POST","access-control-allow-origin":"*","access-control-max-age":"86400","cache-control":"private, no-transform, max-age=929545","content-length":"2778","content-type":"image/png","date":"Sun, 21 Jan 2024 09:41:07 GMT","expires":"Thu, 01 Feb 2024 03:53:32 GMT","last-modified":"Wed, 20 Sep 2023 11:21:46 GMT","server":"Akamai Image Manager","x-check-cacheable":"YES","x-serial":"1231"},"mimeType":"image/png","protocol":"h2","remoteIPAddress":"[2600:140f:9c00:18e::2763]","remotePort":443,"responseTime":1.705830067247854e+12,"securityDetails":{"certificateId":0,"certificateTransparencyCompliance":"compliant","cipher":"AES_256_GCM","encryptedClientHello":false,"issuer":"DigiCert TLS RSA SHA256 2020 CA1","keyExchange":"","keyExchangeGroup":"X25519","protocol":"TLS 1.3","sanList":["*.mmtcdn.com","mmtcdn.com"],"serverSignatureAlgorithm":2052,"signedCertificateTimestampList":[{"hashAlgorithm":"SHA-256","logDescription":"Google \'Argon2024\' log","logId":"EECDD064D5DB1ACEC55CB79DB4CD13A23287467CBCECDEC351485946711FB59B","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"304502200D0F65BCEF39B1565BA630D8578201404421F628FF3E12E8B983833E35B0678A02210092915DEC15BFAF3E74ACF80CF61A27F47F66D22F3AC4FE9245865D9DC36921B6","status":"Verified","timestamp":1.70052397014e+12},{"hashAlgorithm":"SHA-256","logDescription":"DigiCert Yeti2024 Log","logId":"48B0E36BDAA647340FE56A02FA9D30EB1C5201CB56DD2C81D9BBBFAB39D88473","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"304502202EE87ECE4B1D390AFE03557BCD56C254EAEC9D49D271A7AB170657619AC37FAD022100EF6D7BE803809592AA1E62497E7BF9ED10E4CC77BB835B9B639750F51DB22D2E","status":"Verified","timestamp":1.700523970091e+12},{"hashAlgorithm":"SHA-256","logDescription":"Cloudflare \'Nimbus2024\' Log","logId":"DAB6BF6B3FB5B6229F9BC2BB5C6BE87091716CBB51848534BDA43D3048D7FBAB","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"304502201F0BBD8503A19BABB2120D5F4161C7A48BEA74E00428B62E7D8A43F094B85483022100D66FAE5B98AEAABDFA3C0E3CDCF58AD3C519F988BF82FC1268E2DAC1811C372E","status":"Verified","timestamp":1.700523970085e+12}],"subjectName":"*.mmtcdn.com","validFrom":1700438400,"validTo":1732060799},"securityState":"secure","status":200,"statusText":"","timing":{"connectEnd":32.398,"connectStart":2.634,"dnsEnd":2.634,"dnsStart":2.621,"proxyEnd":-1,"proxyStart":-1,"pushEnd":0,"pushStart":0,"receiveHeadersEnd":65.541,"receiveHeadersStart":64.425,"requestTime":80020.796468,"sendEnd":36.28,"sendStart":34.407,"sslEnd":32.392,"sslStart":7.395,"workerFetchStart":-1,"workerReady":-1,"workerRespondWithSettled":-1,"workerStart":-1},"url":"https://imgak.mmtcdn.com/pwa_v3/pwa_hotel_assets/header/mmtLogoWhite.png"},"timestamp":80020.877825,"type":"Image"}},"webview":"9C41AC99D954B28E130D2324C3EA66CE"}', 'timestamp': 1705830067265}
{'level': 'INFO', 'message': '{"message":{"method":"Network.dataReceived","params":{"dataLength":2778,"encodedDataLength":0,"requestId":"21916.5","timestamp":80020.877856}},"webview":"9C41AC99D954B28E130D2324C3EA66CE"}', 'timestamp': 1705830067265}
{'level': 'INFO', 'message': '{"message":{"method":"Network.loadingFinished","params":{"encodedDataLength":3079,"requestId":"21916.5","timestamp":80020.864006}},"webview":"9C41AC99D954B28E130D2324C3EA66CE"}', 'timestamp': 1705830067265}
{'level': 'INFO', 'message': '{"message":{"method":"Network.responseReceived","params":{"frameId":"9C41AC99D954B28E130D2324C3EA66CE","hasExtraInfo":true,"loaderId":"86836D729FDC3171EA9C611C9DA5888C","requestId":"21916.7","response":{"alternateProtocolUsage":"unspecifiedReason","connectionId":147,"connectionReused":true,"encodedDataLength":264,"fromDiskCache":false,"fromPrefetchCache":false,"fromServiceWorker":false,"headers":{"access-control-allow-credentials":"false","access-control-allow-headers":"*","access-control-allow-methods":"GET,POST","access-control-allow-origin":"*","access-control-max-age":"86400","cache-control":"private, no-transform, max-age=1191256","content-length":"18243","content-type":"image/avif","date":"Sun, 21 Jan 2024 09:41:07 GMT","expires":"Sun, 04 Feb 2024 04:35:23 GMT","last-modified":"Tue, 17 Oct 2023 11:22:16 GMT","server":"Akamai Image Manager"},"mimeType":"image/avif","protocol":"h2","remoteIPAddress":"[2600:140f:9c00:18e::2763]","remotePort":443,"responseTime":1.705830067247984e+12,"securityDetails":{"certificateId":0,"certificateTransparencyCompliance":"compliant","cipher":"AES_256_GCM","encryptedClientHello":false,"issuer":"DigiCert TLS RSA SHA256 2020 CA1","keyExchange":"","keyExchangeGroup":"X25519","protocol":"TLS 1.3","sanList":["*.mmtcdn.com","mmtcdn.com"],"serverSignatureAlgorithm":2052,"signedCertificateTimestampList":[{"hashAlgorithm":"SHA-256","logDescription":"Google \'Argon2024\' log","logId":"EECDD064D5DB1ACEC55CB79DB4CD13A23287467CBCECDEC351485946711FB59B","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"304502200D0F65BCEF39B1565BA630D8578201404421F628FF3E12E8B983833E35B0678A02210092915DEC15BFAF3E74ACF80CF61A27F47F66D22F3AC4FE9245865D9DC36921B6","status":"Verified","timestamp":1.70052397014e+12},{"hashAlgorithm":"SHA-256","logDescription":"DigiCert Yeti2024 Log","logId":"48B0E36BDAA647340FE56A02FA9D30EB1C5201CB56DD2C81D9BBBFAB39D88473","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"304502202EE87ECE4B1D390AFE03557BCD56C254EAEC9D49D271A7AB170657619AC37FAD022100EF6D7BE803809592AA1E62497E7BF9ED10E4CC77BB835B9B639750F51DB22D2E","status":"Verified","timestamp":1.700523970091e+12},{"hashAlgorithm":"SHA-256","logDescription":"Cloudflare \'Nimbus2024\' Log","logId":"DAB6BF6B3FB5B6229F9BC2BB5C6BE87091716CBB51848534BDA43D3048D7FBAB","origin":"Embedded in certificate","signatureAlgorithm":"ECDSA","signatureData":"304502201F0BBD8503A19BABB2120D5F4161C7A48BEA74E00428B62E7D8A43F094B85483022100D66FAE5B98AEAABDFA3C0E3CDCF58AD3C519F988BF82FC1268E2DAC1811C372E","status":"Verified","timestamp":1.700523970085e+12}],"subjectName":"*.mmtcdn.com","validFrom":1700438400,"validTo":1732060799},"securityState":"secure","status":200,"statusText":"","timing":{"connectEnd":-1,"connectStart":-1,"dnsEnd":-1,"dnsStart":-1,"proxyEnd":-1,"proxyStart":-1,"pushEnd":0,"pushStart":0,"receiveHeadersEnd":35.667,"receiveHeadersStart":34.497,"requestTime":80020.826547,"sendEnd":6.206,"sendStart":5.763,"sslEnd":-1,"sslStart":-1,"workerFetchStart":-1,"workerReady":-1,"workerRespondWithSettled":-1,"workerStart":-1},"url":"https://imgak.mmtcdn.com/pwa_v3/pwa_header_assets/loginPersuassionRoad.jpg"},"timestamp":80020.878241,"type":"Image"}},"webview":"9C41AC99D954B28E130D2324C3EA66CE"}', 'timestamp': 1705830067265}
{'level': 'INFO', 'message': '{"message":{"method":"Network.dataReceived","params":{"dataLength":18243,"encodedDataLength":0,"requestId":"21916.7","timestamp":80020.878257}},"webview":"9C41AC99D954B28E130D2324C3EA66CE"}', 'timestamp': 1705830067266}
{'level': 'INFO', 'message': '{"message":{"method":"Network.loadingFinished","params":{"encodedDataLength":18525,"requestId":"21916.7","timestamp":80020.866001}},"webview":"9C41AC99D954B28E130D2324C3EA66CE"}', 'timestamp': 1705830067266}
Now, most of the network response results are in the above output. We need to parse the output and retrieve the values we need.
Capture Network call response
We can capture information about network call responses by listening to the “Network.responseReceived” event. This event is fired whenever an HTTP response is received.
The event is recorded in the log_entries as a “method“. We can obtain the method name and use an if statement to verify if it matches with “Network.responseReceived”. If it does, we can then retrieve all the necessary details like status code, headers, etc.
message_obj = json.loads(entry.get("message"))
message = message_obj.get("message")
method = message.get("method")
if method == 'Network.responseReceived':
#get details here
Below is one example JSON of the “Network.responseReceived” event
{
"message":{
"method":"Network.responseReceived",
"params":{
"frameId":"F27E641923D2D8E6E13D90E187C7C047",
"hasExtraInfo":true,
"loaderId":"C7E38B506DF3479116F80D5D4430DA68",
"requestId":"4992.39",
"response":{
"alternateProtocolUsage":"unspecifiedReason",
"connectionId":64,
"connectionReused":true,
"encodedDataLength":21093,
"fromDiskCache":false,
"fromPrefetchCache":false,
"fromServiceWorker":false,
"headers":{
"akamai-cache-status":"Hit from child",
"cache-control":"max-age=86400",
"content-encoding":"gzip",
"content-length":"20807",
"content-type":"application/javascript",
"date":"Sun, 21 Jan 2024 05:59:40 GMT",
"etag":"\"00cb6040d049d396de005ea66dd3916043ea887156b97f740a5256a2daeaf1dd\"",
"last-modified":"Tue, 07 Nov 2023 18:44:09 GMT",
"stored-attribute-sha-checksum":"d96aa78f2d55331a8b70741c7230d2c2fd54310736a8454d53ac391cea813755",
"vary":"Accept-Encoding"
},
"mimeType":"application/javascript",
"protocol":"h2",
"remoteIPAddress":"[2600:1417:56::174c:9d12]",
"remotePort":443,
"responseTime":1705816780653.71,
"securityDetails":{
"certificateId":0,
"certificateTransparencyCompliance":"compliant",
"cipher":"AES_256_GCM",
"encryptedClientHello":false,
"issuer":"DigiCert TLS RSA SHA256 2020 CA1",
"keyExchange":"",
"keyExchangeGroup":"X25519",
"protocol":"TLS 1.3",
"sanList":[
"*.makemytrip.com",
"makemytrip.com"
],
"serverSignatureAlgorithm":1027,
"signedCertificateTimestampList":[
{
"hashAlgorithm":"SHA-256",
"logDescription":"Google 'Argon2024' log",
"logId":"EECDD064D5DB1ACEC55CB79DB4CD13A23287467CBCECDEC351485946711FB59B",
"origin":"Embedded in certificate",
"signatureAlgorithm":"ECDSA",
"signatureData":"3045022100EC312CCF674E20F1E73DEEAD0C16D35879C0B148C2A8C05D2533754762BEBC4A0220447913CF30CBCD5583382E53420746E73F5D90A64F8ACB89DF97AE8D48521D5A",
"status":"Verified",
"timestamp":1674803490133.0
},
{
"hashAlgorithm":"SHA-256",
"logDescription":"DigiCert Nessie2024 Log",
"logId":"73D99E891B4C9678A0207D479DE6B2C61CD0515E71192A8C6B80107AC17772B5",
"origin":"Embedded in certificate",
"signatureAlgorithm":"ECDSA",
"signatureData":"30450221008E6CF469BF71E3E1AA3CB3C7A7D129DFCBC24336114C89A87399DD57BEAC22B002205E78B7776A9CF6F1504D4B44F19E64B70263CDE4FE34E25C3D0C2FCC461454A3",
"status":"Verified",
"timestamp":1674803490234.0
},
{
"hashAlgorithm":"SHA-256",
"logDescription":"DigiCert Yeti2024 Log",
"logId":"48B0E36BDAA647340FE56A02FA9D30EB1C5201CB56DD2C81D9BBBFAB39D88473",
"origin":"Embedded in certificate",
"signatureAlgorithm":"ECDSA",
"signatureData":"3045022100D10A2155226963B2E6A250EB6A3CF654458DD3CDDF72F38EE140F89F1DA713F0022036A5F262F8A7114D42BFCA5C4A01BF1F1045B1BFA472A55D7D9950A6E66BE7A0",
"status":"Verified",
"timestamp":1674803490183.0
}
],
"subjectName":"*.makemytrip.com",
"validFrom":1674777600,
"validTo":1706659199
},
"securityState":"secure",
"status":200,
"statusText":"",
"timing":{
"connectEnd":-1,
"connectStart":-1,
"dnsEnd":-1,
"dnsStart":-1,
"proxyEnd":-1,
"proxyStart":-1,
"pushEnd":0,
"pushStart":0,
"receiveHeadersEnd":14.129,
"receiveHeadersStart":13.621,
"requestTime":66734.25293,
"sendEnd":0.618,
"sendStart":0.352,
"sslEnd":-1,
"sslStart":-1,
"workerFetchStart":-1,
"workerReady":-1,
"workerRespondWithSettled":-1,
"workerStart":-1
},
"url":"https://www.makemytrip.com/_sec/cp_challenge/sec-cpt-4-1.js"
},
"timestamp":66734.297152,
"type":"Script"
}
},
"webview":"F27E641923D2D8E6E13D90E187C7C047"
}
Getting Response code
So, the path to extract the URL and status code is –
response_url - message -> params -> response -> url
response_code - message -> params -> response -> status
response_url = message.get('params',{}).get('response',{}).get('url','')
response_code = message.get('params',{}).get('response',{}).get('status','')
Whole code
from selenium import webdriver
import json
#enable logging
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.makemytrip.com")
# Capture network log entries
log_entries = driver.get_log("performance")
for entry in log_entries:
try:
message_obj = json.loads(entry.get("message"))
message = message_obj.get("message")
method = message.get("method")
if method == 'Network.responseReceived':
response_url = message.get('params',{}).get('response',{}).get('url','')
response_code = message.get('params',{}).get('response',{}).get('status','')
if "makemytrip" in response_url:
print("Response URL: " + response_url+ ", Response Code: " + str(response_code))
except Exception as e:
print(e)
Getting Response Headers
The path to extract the response headers is –
response_headers - message -> params -> response -> headers
response_url = message.get('params',{}).get('response',{}).get('url','')
response_headers = message.get('params',{}).get('response',{}).get('headers',{})
Whole Code
from selenium import webdriver
import json
#enable logging
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.makemytrip.com")
# Capture network log entries
log_entries = driver.get_log("performance")
for entry in log_entries:
try:
message_obj = json.loads(entry.get("message"))
message = message_obj.get("message")
method = message.get("method")
if method == 'Network.responseReceived':
response_url = message.get('params',{}).get('response',{}).get('url','')
response_headers = message.get('params',{}).get('response',{}).get('headers',{})
if "makemytrip" in response_url:
print("Response URL: " + response_url+ ", Response Headers: " + str(response_headers))
except Exception as e:
print(e)
Getting Response Body
It is evident from the JSON of the “Network.responseReceived” method that the response body is not included. So, how can we retrieve the response body for a network call?
Well, it would be a 2-step process –
- Retrieving requestId from the “Network.responseReceived” JSON.
- and then use the Network.getResponseBody method, which accepts requestId as a parameter and will return the corresponding response body.
Retrieving requestId
requestId - message -> params -> requestId
request_id = message.get('params', {}).get('requestId', '')
Getting response body
First, we would need to enable the network tracking using execute_cdp_cmd() method and then use the same method to retrieve the response body.
driver.execute_cdp_cmd('Network.enable', {})
response = driver.execute_cdp_cmd('Network.getResponseBody',{'requestId':request_id})
response_body = response.get('body','')
Whole Code
from selenium import webdriver
import json
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('detach',True)
chrome_options.add_argument('--enable-logging')
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.makemytrip.com")
# enable network tracking
driver.execute_cdp_cmd('Network.enable', {})
# Capture network log entries
log_entries = driver.get_log("performance")
for entry in log_entries:
try:
message_obj = json.loads(entry.get("message"))
message = message_obj.get("message")
method = message.get("method")
if method == 'Network.responseReceived':
response_url = message.get('params',{}).get('response',{}).get('url','')
request_id = message.get('params', {}).get('requestId', '')
if "makemytrip" in response_url:
response = driver.execute_cdp_cmd('Network.getResponseBody',{'requestId':request_id})
response_body = response.get('body','')
print("Response URL: " + response_url+ ", Response Body: " + str(response_body))
except Exception as e:
print(e)
We hope that you like the article. If you have any doubts or concerns, please write to us in the comments or mail us at admin@codekru.com.