Btukfyl

I am scraping this website: https://www.misoenergy.org/markets-and-operations/market-reports/market-report-archives/#nt=%2FMarketReportType%3ABids%2FMarketReportName%3AArchived%20Cleared%20Bids%20%20(zip)&t=10&p=0&s=FileName&sd=desc

And try to download all the zip files from the table. However, I can not locate the table from the 'soup'. It returns nothing.

req = Request(

    'https://www.misoenergy.org/markets-and-operations/market-reports/market-report-archives/#nt=%2FMarketReportType%3ABids%2FMarketReportName%3AArchived%20Cleared%20Bids%20%20(zip)&t=10&p=0&s=FileName&sd=desc',

     headers={'User-Agent': 'Mozilla/5.0'})

page = urlopen(req).read()

soup = BeautifulSoup(page, "html.parser")

tables = soup.find('div', class_='table table-bordered docnav-metadata dataTable no-footer')

as stated, you need something like selenium to load the page as it's dynamic. You'll also need to let it wait to load to get the table.

NOTE: I used time.sleep() for the wait, but I have read that is not the best solution. The suggestion is to use WebDriverWait but I'm still in the pr0cess of understanding how that would work, so will update this once I play around. In the meantime, this should get you started.

import bs4 

from selenium import webdriver

import time



driver = webdriver.Chrome()

driver.get('https://www.misoenergy.org/markets-and-operations/market-reports/market-report-archives/#nt=%2FMarketReportType%3ABids%2FMarketReportName%3AArchived%20Cleared%20Bids%20%20(zip)&t=10&p=0&s=FileName&sd=desc')





time.sleep(5)    



html = driver.page_source

soup = bs4.BeautifulSoup(html,'html.parser')



tables = soup.findAll('table', {'class':'table table-bordered docnav-metadata dataTable no-footer'})

This worked for me with WebDriverWait:

import bs4 

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.wait import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC



driver = webdriver.Chrome()

driver.get('https://www.misoenergy.org/markets-and-operations/market-reports/market-report-archives/#nt=%2FMarketReportType%3ABids%2FMarketReportName%3AArchived%20Cleared%20Bids%20%20(zip)&t=10&p=0&s=FileName&sd=desc')



WebDriverWait(driver, 10).until(

        EC.presence_of_element_located((By.CSS_SELECTOR, "table.table-bordered.docnav-metadata.dataTable.no-footer")))



html = driver.page_source

soup = bs4.BeautifulSoup(html,'html.parser')



tables = soup.findAll('table', {'class':'table table-bordered docnav-metadata dataTable no-footer'})

To fetch the tabular content from that webpage using Requests-HTML library, you can try the following script:

import requests_html



link = "https://www.misoenergy.org/markets-and-operations/market-reports/market-report-archives/#nt=%2FMarketReportType%3ABids%2FMarketReportName%3AArchived%20Cleared%20Bids%20%20(zip)&t=10&p=0&s=FileName&sd=desc"



with requests_html.HTMLSession() as session:

    r = session.get(link)

    r.html.render(sleep=5,timeout=8)

    for items in r.html.find("table.dataTable tr.desktop-row"):

        data = [item.text for item in items.find("td")]

        print(data)