Regex to capture to specific percent/decimals

dtrinh Source

I am trying to scrape for interest rates in multiple sites. The data is pretty unstructured but close enough in form. What I want to capture:

x.xx% to xx.xx%

Examples of what the data looks like:

All loans made by WebBank, Member FDIC. Your actual rate depends upon credit score, loan amount, loan term, and credit usage & history. The APR ranges from 5.98% to 35.89%. For example, you could receive a loan of $6,000 with an interest rate of 7.99% and a 5.00% origination fee of $300 for an APR of 11.51%. In this example, you will receive $5,700 and will make 36 monthly payments of $187.99. The total amount repayable will be $6,767.64. Your APR will be determined based on your credit at time of application. The origination fee ranges from 1% to 6% and the average origination fee is 5.49% as of Q1 2017. There is no down payment and there is never a prepayment penalty. Closing of your loan is contingent upon your agreement of all the required agreements and disclosures on the www.lendingclub.com website. All loans via LendingClub have a minimum repayment term of 36 months or longer.

3.09% – 14.24%*

Fixed rates: 6.99% to 24.99% APR Lock in your rate. Your monthly payment will never change.

I've bolded what I wanted to capture. My current regex looks like this:

(re.findall('(?i)(\d\.\d\d% (?:to|-) \d\d\.\d\d%)

The actual quote looks like:

plcompetitors = ['https://www.lendingclub.com/loans/personal-loans',
                'https://www.marcus.com/us/en/personal-loans',
                'https://www.discover.com/personal-loans/',
                'https://www.lightstream.com/',
                'https://www.prosper.com/']

#cycle through links in array until it finds APR rates/fixed or variable using regex
for link in plcompetitors:
    cdate = datetime.date.today()
    l = r.get(link)
    l.encoding = 'utf-8'
    data = l.text
    soup = bs(data, 'html.parser')
    paragraph = soup.find_all(text=re.compile('[0-9]%'))
    for n in paragraph:
        matches = []
        matches.extend(re.findall('(?i)(\d\.\d\d% (?:to|-) \d\d\.\d\d%)', n.string))
        matches.append(cdate.isoformat())
        matches.append(link)
        print(matches)
    paragraph.append(cdate.isoformat())
    paragraph.append(link)

New Output:

['5.98% to 35.89%', '2018-06-22', 'https://www.lendingclub.com/loans/personal-loans']
['2018-06-22', 'https://www.lendingclub.com/loans/personal-loans']
['6.99% to 24.99%', '6.99% to 24.99%', '6.99% to 24.99%', '6.99% to 24.99%', '2018-06-22', 'https://www.marcus.com/us/en/personal-loans']
['2018-06-22', 'https://www.marcus.com/us/en/personal-loans']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['6.99% to 24.99%', '2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.discover.com/personal-loans/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.lightstream.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
['2018-06-22', 'https://www.prosper.com/']
pythonregexpython-3.xbeautifulsoup

Answers

answered 4 weeks ago Inquisitor01 #1

Edit: In light of your comment Run the following in Python3 which should processes your example string in ASCII at default:

Input

import re

input = '''All loans made by WebBank, Member FDIC. Your actual rate depends upon credit score, loan amount, loan term, and credit usage & history. The APR ranges from 5.98% to 35.89%. For example, you could receive a loan of $6,000 with an interest rate of 7.99% and a 5.00% origination fee of $300 for an APR of 11.51%. In this example, you will receive $5,700 and will make 36 monthly payments of $187.99. The total amount repayable will be $6,767.64. Your APR will be determined based on your credit at time of application. The origination fee ranges from 1% to 6% and the average origination fee is 5.49% as of Q1 2017. There is no down payment and there is never a prepayment penalty. Closing of your loan is contingent upon your agreement of all the required agreements and disclosures on the www.lendingclub.com website. All loans via LendingClub have a minimum repayment term of 36 months or longer.

3.09% – 14.24%*

Fixed rates: 6.99% to 24.99% APR Lock in your rate. Your monthly payment will never change.'''
#Non-specific regex (I'm cheating)
output = re.findall('[\d]{1,3}\.[\d]+%[\S\s]{0,5}[\d]{1,3}\.[\d]+%', input)
print('output:')
print(output)

#More specific -- you can edit this in several ways
output_1 = re.findall('[\d]{1,3}\.[\d]+%[to\-\s]+[\d]{1,3}\.[\d]+%', input)
print('\noutput_1:')
print(output_1)

#What you need if you copy+paste from Stack into Python2.7.X
output_2 = re.findall('[\d]{1,3}\.[\d]+%[\s]*[to|\-|\xe2\x80\x93]+[\s]*[\d]{1,3}\.[\d]+%', input)
print('\noutput_2 (Python2.X):')
print(output_2)

Output

output:
['5.98% to 35.89%', '3.09% - 14.24%', '6.99% to 24.99%']

output_1:
['5.98% to 35.89%', '3.09% - 14.24%', '6.99% to 24.99%']

output_2 (Python2.X)::
['5.98% to 35.89%', '3.09% \xe2\x80\x93 14.24%', '6.99% to 24.99%']

answered 3 weeks ago Wiktor Stribiżew #2

The paragraph = soup.find_all(text=re.compile('(?i)(\d\.\d\d% (?:to|-) \d\d\.\d\d%)')) line gets all the nodes with values matching your pattern. You need to actually extract the matches from these paragraphs.

Use something like

matches=[]
for n in paragraph:
    matches.extend(re.findall(pattern, n.string))

As for the pattern itself, you may use

(?i)\d+(?:\.\d+)?%\s*(?:to|-)\s*\d+(?:\.\d+)?%

See the regex demo. Details:

  • (?i) - case insensitive maching is ON
  • \d+(?:\.\d+)? - 1+ digits that is optionally followed with . and 1+ digits
  • % - a % sign
  • \s* - 0+ whitespaces
  • (?:to|-) - to or -
  • \s*\d+(?:\.\d+)?% - see above (in short, whitespace(s), an int or float value followed with %).

comments powered by Disqus