dealing with async

charts/templates/charts/fr.html

@@ -221,7 +221,7 @@
                   <div class="icon-box mt-4 mt-xl-0">
                     <i class="bx bx-stats"></i>
                     <h4>Monero a une faible inflation</h4>
-                    <p>L'inflation annuelle actuelle n'est que de {{ now_inflation }} et diminue constamment jusqu'à ce
+                    <p>L'inflation annuelle actuelle n'est que de {{ inflation }} et diminue constamment jusqu'à ce
                       que les émissions "de queue" commencent vers 2022. Il n'y avait pas de
                       prémine ou ICO et toutes les pièces ont été extraites équitablement depuis 2014.
                     </p>
@@ -529,7 +529,7 @@
                     de valider toutes les transactions sur le réseau.
                     En faisant cela, toute cette puissance de traitement
                     est utilisée pour garantir une émission équitable de nouvelles pièces
-                    (ce qui provoque une inflation annuelle décroissante de {{now_inflation}}) et pour
+                    (ce qui provoque une inflation annuelle décroissante de {{inflation}}) et pour
                     protéger votre argent
                     des pirates informatiques. Monero a une inflation
                     décroissante qui finira par tendre vers zéro, et parce qu'une faible
@@ -616,8 +616,8 @@
                     d'une forme de revenu financier qu'ils reçoivent pour leur service.
                     Depuis au
                     au début (2014), il n'y avait pas de pièces en
-                    circulation, l'inflation était élevée. Maintenant qu'il y a {{now_units}}
+                    circulation, l'inflation était élevée. Maintenant qu'il y a {{supply}}
-                    unités en circulation, l'inflation annuelle n'est que de {{now_inflation}}.
+                    unités en circulation, l'inflation annuelle n'est que de {{inflation}}.
                     Le paiement par bloc diminue avec le temps, de même
                     que le taux d'inflation annuel diminuera lentement vers zéro. La
                     demande d'argent dur fongible, numérique et axé sur la confidentialité

charts/templates/charts/index.html

@@ -193,7 +193,7 @@
                 <div class="icon-box mt-4 mt-xl-0">
                   <i class="bx bx-stats"></i>
                   <h4>Monero has low inflation</h4>
-                  <p>The current annual inflation is only {{ now_inflation }} and is constantly decreasing until "tail"
+                  <p>The current annual inflation is only {{ inflation }} and is constantly decreasing until "tail"
                     emission kicks in around 2022. There was no premine
                     or ICOs and all coins have been mined fairly since 2014.
                   </p>
@@ -417,7 +417,7 @@
                 Many people leave their computers "mining" it, which means they lend their PC's processing power in
                 order to validate all transactions on the network.
                 By doing this, all this processing power is used to guarantee fair emission of new coins (which causes
-                the decreasing annual inflation of {{ now_inflation }}) and to protect your money
+                the decreasing annual inflation of {{ inflation }}) and to protect your money
                 from hackers. Monero has a decreasing inflation that will eventually tend to zero, and because low
                 inflation leads to scarcity, it works as a store of value, just like Gold or Bitcoin.
                 Differently from Bitcoin, though, Monero has full privacy, which means it hides your balance and your
@@ -473,7 +473,7 @@
                 receive coins based on how much work they put into validating and defending the network, so it is a form
                 of financial revenue they get for their service. Since at the
                 beginning (2014) there were no coins in circulation, inflation was high. Now that there are {{
-                now_units}} units in circulation, the annual inflation is just {{ now_inflation }}.
+                supply}} units in circulation, the annual inflation is just {{ inflation }}.
                 The payout per block decreases over time, and so will the the annual inflation rate slowly decrease
                 towards zero. The demand for fungible, digital, privacy-oriented hard money will likely skyrocket
                 in the next few years and outpace its inflation, so investors expect a considerable rise in prices.

charts/templates/charts/pt.html

@@ -204,7 +204,7 @@
                     <div class="icon-box mt-4 mt-xl-0">
                       <i class="bx bx-stats"></i>
                       <h4>Monero tem inflação baixa</h4>
-                      <p>A inflação anual atual é de apenas {{ now_inflation }} e está diminuindo constantemente até que a emissão "residual" comece por volta de 2022. 
+                      <p>A inflação anual atual é de apenas {{ inflation }} e está diminuindo constantemente até que a emissão "residual" comece por volta de 2022. 
                         Não houve moedas mineradas de maneira desleal em seu início, sendo todas mineradas de forma justa desde 2014. 
                       </p>
                     </div>
@@ -422,7 +422,7 @@
                   Nunca houve uma pré-mineração em Monero. Isso significa que ninguém controla grandes quantidades de moedas, uma vez que todas as moedas geradas 
                   sempre foram distribuídas para os mineradores que trabalhassem honestamente. Os mineradores recebem moedas com base em quanto trabalho eles colocam 
                   em validar e defender a rede, portanto, é uma forma de receita financeira que eles obtêm por seus serviços. No início (2014) não havia moedas em 
-                  circulação, a inflação era elevada. Hoje existem {{ now_units }} unidades em circulação, a inflação anual é de apenas {{ now_inflation}}. 
+                  circulação, a inflação era elevada. Hoje existem {{ supply }} unidades em circulação, a inflação anual é de apenas {{ inflation}}. 
                   O pagamento por bloco diminui ao longo do tempo, e também a taxa de inflação anual diminuirá lentamente até chegar a zero. 
                   A demanda por dinheiro fungível, digital e voltado para a privacidade provavelmente disparará nos próximos anos e ultrapassará a inflação, fazendo com que 
                   provavelmente haja uma apreciação desse ativo.

charts/views.py

@@ -34,90 +34,70 @@ locale.setlocale(locale.LC_ALL, 'en_US.utf8')
 # This loads Reddit stats about a subreddit
 api = PushshiftAPI() 
 
+##########################################
+#   Asynchronous get block data
+########################################## 
+async def get_block_data(session, block: str):
+    url = 'https://xmrchain.net/api/block/' + block
+    async with session.get(url) as res:
+        data = await res.read()
+        data = json.loads(data)
+        data['provider'] = 'block_data_xmrchain'
+        if res.status < 299:
+            data['success'] = True
+        else:
+            data['success'] = False
+        return data
+
+##########################################
+#   Asynchronous get network data
+########################################## 
+async def get_network_data(session, block: str):
+    url = 'https://xmrchain.net/api/networkinfo'
+    async with session.get(url) as res:
+        data = await res.read()
+        data = json.loads(data)
+        data['provider'] = 'network_data_xmrchain'
+        if res.status < 299:
+            data['success'] = True
+        else:
+            data['success'] = False
+        return data
+
+##########################################
+#   Asynchronous get coinmarketcap data
+########################################## 
+async def get_coinmarketcap_data(session, symbol: str, convert: str):
+    with open("settings.json") as file:
+        data = json.load(file)
+        file.close()
+
+    url = data["metrics_provider"][0]["price_url_old"] + symbol
+    parameters = {'convert': convert,}
+    headers = {'Accepts': 'application/json', data["metrics_provider"][0]["api_key_name"]: data["metrics_provider"][0]["api_key_value"],}
+
+    async with session.get(url, headers=headers, params=parameters) as res:
+        data = await res.read()
+        data = json.loads(data)
+        data['provider'] = 'coinmarketcap'
+        if res.status < 299:
+            try:
+                if data['data'][symbol.upper()]['cmc_rank']:
+                    data['success'] = True
+                else:
+                    data['success'] = False
+            except:
+                data['success'] = False
+        else:
+            data['success'] = False
+        return data
+
 ###########################################
 # Useful functions for admins
 ###########################################
 
 # Add manually a new entrance for coin
 # To be used when there's a problem with the API
-
-async def get_monero_data():
-    data = get_latest_price('xmr')
-    data_aux = get_latest_price('btc')
-    if not(data):
-        print('error updating rank')
-        return False
-    else:
-        name = 'xmr'
-        date_aux = date.today() - timedelta(1)
-        date_aux = datetime.datetime.strftime(date_aux, '%Y-%m-%d')
-        date_aux2 = date.today() - timedelta(2)
-        date_aux2 = datetime.datetime.strftime(date_aux2, '%Y-%m-%d')
-        supply = int(data['data']['XMR']['circulating_supply'])
-        priceusd = float(data['data']['XMR']['quote']['USD']['price'])
-        pricebtc = float(data['data']['XMR']['quote']['USD']['price'])/float(data_aux['data']['BTC']['quote']['USD']['price'])
-        try:
-            coin = Coin.objects.filter(name='xmr').get(date=date_aux)
-        except:
-            return False
-        inflation = 100*365*(float(supply - coin.supply))/float(coin.supply)
-        stocktoflow = (100/inflation)**1.65 
-
-        url = 'https://xmrchain.net/api/networkinfo'
-        response = requests.get(url)
-        data = json.loads(response.text)
-        difficulty = int(data['data']['difficulty'])
-        hashrate = int(data['data']['hash_rate'])
-        height = int(data['data']['height'])
-
-        ##########################################
-        #   Asynchronous
-        ##########################################
-        actions = []
-        my_timeout = aiohttp.ClientTimeout(
-            total=5,
-            sock_connect=10, 
-            sock_read=5 
-        )
-        client_args = dict(
-            trust_env=True,
-            timeout=my_timeout
-        )
-
-        async with aiohttp.ClientSession(**client_args) as session:  
-            for count in range(1, 1500):
-                block = str(height - count)
-                print(block)
-                actions.append(asyncio.ensure_future(get_block(session, block)))
-
-            try:
-                responses = await asyncio.gather(*actions, return_exceptions=True)
-            except asyncio.exceptions.TimeoutError:
-                print('Timeout!')
-
-            for response in responses:
-                try:
-                    print(datetime.datetime.strptime(response['data']['timestamp_utc'], '%Y-%m-%d %H:%M:%S'))
-                except:
-                    print('error')
-
-    return True
-
-##########################################
-#   Asynchronous get block data
-########################################## 
-async def get_block(session, block: str):
-    url = 'https://xmrchain.net/api/block/' + block
-    async with session.get(url) as res:
-        if res.status < 299:
-            data = await res.read()
-            data = json.loads(data)
-            return data
-        else:
-            return False
-
-
-
 @login_required
 def add_coin(request):
     if request.user.username != "Administrador" and request.user.username != "Morpheus":
@@ -1126,13 +1106,12 @@ def update_dominance(data):
         else:
             print('spreadsheet already with the latest data')
             return False
-
-    #print('updated')
     return data
 
 # Get latest rank value and update
-def update_rank():
+def update_rank(data=None):
-    data = get_latest_price('xmr')
+    if not(data):
+        data = get_latest_price('xmr')
     if not(data):
         print('error updating rank')
         return False
@@ -1163,7 +1142,6 @@ def update_rank():
             print('spreadsheet already with the latest data')
             return data
 
-    #print('updated')
     return data
 
 # Load Reddit api to check if there are new followers
@@ -1635,56 +1613,61 @@ def update_p2pool():
 async def index(request):
     if request.user.username != "Administrador" and request.user.username != "Morpheus":
         update_visitors(True)
+
     ###################################################################################
 
-    data = get_latest_price('xmr')
+    dt = datetime.datetime.now(timezone.utc).timestamp()
-    data_aux = get_latest_price('btc')
+    symbol = 'xmr'
-    if not(data):
+
-        print('error updating')
+    ###################################################################################
-        return False
+
-    else:
+    coins = Coin.objects.filter(name=symbol).order_by('-date')
-        name = 'xmr'
+    supply = 0
-        yesterday = date.today() - timedelta(1)
+    if coins:
+        for coin in coins:
+            if coin.supply > 0:
+                supply = coin.supply
+                inflation = coin.inflation
+                break
+    if supply == 0:
+        message = 'Website under maintenance. Check back in a few minutes'
+        context = {'message': message}
+        return render(request, 'charts/maintenance.html', context)
+
+    ###################################################################################
+
+    url = 'https://xmrchain.net/api/networkinfo'
+    response = requests.get(url)
+    data = json.loads(response.text)
+    height = int(data['data']['height'])
+    difficulty = int(data['data']['difficulty'])
+    hashrate = int(data['data']['hash_rate'])
+    blocksize = 0
+    name = 'xmr'
+    yesterday = date.today() - timedelta(1)
+
+    ##########################################
+    #   Asynchronous
+    ##########################################
+    if yesterday > coin.date:
         yesterday = datetime.datetime.strftime(yesterday, '%Y-%m-%d')
-        supply = int(data['data']['XMR']['circulating_supply'])
-        priceusd = float(data['data']['XMR']['quote']['USD']['price'])
-        pricebtc = float(data['data']['XMR']['quote']['USD']['price'])/float(data_aux['data']['BTC']['quote']['USD']['price'])
-        try:
-            coins = Coin.objects.filter(name='xmr').order_by('-date')
-            for coin in coins:
-                if coin.supply > 0:
-                    break
-        except:
-            return False
-        inflation = 100*365*(float(supply - coin.supply))/float(coin.supply)
-        stocktoflow = (100/inflation)**1.65 
-
-        url = 'https://xmrchain.net/api/networkinfo'
-        response = requests.get(url)
-        data = json.loads(response.text)
-        difficulty = int(data['data']['difficulty'])
-        hashrate = int(data['data']['hash_rate'])
-        height = int(data['data']['height'])
-
-        ##########################################
-        #   Asynchronous
-        ##########################################
         actions = []
         my_timeout = aiohttp.ClientTimeout(
-            total=5,
+            total=10,
             sock_connect=10, 
-            sock_read=5 
+            sock_read=10 
         )
         client_args = dict(
             trust_env=True,
             timeout=my_timeout
         )
 
-        blocksize = 0
         async with aiohttp.ClientSession(**client_args) as session:  
-            for count in range(1, 1000):
+            for count in range(1, 1400):
                 block = str(height - count)
-                actions.append(asyncio.ensure_future(get_block(session, block)))
+                actions.append(asyncio.ensure_future(get_block_data(session, block)))
+            actions.append(asyncio.ensure_future(get_coinmarketcap_data(session, 'xmr', 'USD')))
+            actions.append(asyncio.ensure_future(get_coinmarketcap_data(session, 'xmr', 'BTC')))
 
             try:
                 responses = await asyncio.gather(*actions, return_exceptions=True)
@@ -1696,27 +1679,50 @@ async def index(request):
             txs = 0
             revenue = 0
             fees = 0
+            priceusd = 0
+            pricebtc = 0
             for response in responses:
-                try:
+                if response:
-                    date_aux = response['data']['timestamp_utc'].split(' ')[0]
+                    try:
-                    if date_aux == yesterday:
+                        if response['provider'] == 'block_data_xmrchain':
-                        success += 1
+                            date_aux = response['data']['timestamp_utc'].split(' ')[0]
-                        blocksize += int(response['data']['size'])
+                            if date_aux == yesterday:
-                        for tx in response['data']['txs']:
+                                success += 1
-                            if tx['coinbase']:
+                                try:
-                                revenue += int(tx['xmr_outputs'])
+                                    blocksize += int(response['data']['size'])
-                            else:
+                                    for tx in response['data']['txs']:
-                                txs += 1
+                                        if tx['coinbase']:
-                                fees += int(tx['tx_fee'])
+                                            revenue += int(tx['xmr_outputs'])
-                                revenue += int(tx['tx_fee'])
+                                        else:
-                except:
+                                            txs += 1
+                                            fees += int(tx['tx_fee'])
+                                            revenue += int(tx['tx_fee'])
+                                except:
+                                    errors += 1
+
+                        if response['provider'] == 'coinmarketcap':
+                            try:
+                                priceusd = float(response['data']['XMR']['quote']['USD']['price'])
+                                update_rank(response)    
+                                update_dominance(response)
+                            except:
+                                try:
+                                    pricebtc = float(response['data']['XMR']['quote']['BTC']['price'])
+                                except:
+                                    errors += 1
+                    except:
+                        errors += 1
+                else:
                     errors += 1
+
             blocksize = blocksize/success
             revenue = float(revenue)/10**12
             fees = float(fees)/10**12
             inflation = 100*365*(revenue)/float(coin.supply)
+            stocktoflow = (100/inflation)**1.65 
             supply = coin.supply + revenue
                     
+            print('Name: ' + name)
             print('Date: ' + str(yesterday))
             print('Success: ' + str(success))
             print('Errors: ' + str(errors))
@@ -1731,42 +1737,33 @@ async def index(request):
             print('Priceusd: ' + str(priceusd))
             print('Pricebtc: ' + str(pricebtc))
             print('Supply: ' + str(supply))
-            print('Inflation: ' + str(inflation))
-            print('Inflation: ' + str(inflation))
 
+            try:
+                coin = Coin()
+                coin.name = name
+                coin.date = datetime.datetime.strptime(yesterday, '%Y-%m-%d')
+                coin.blocksize = blocksize
+                coin.transactions = txs
+                coin.revenue = revenue
+                coin.fee = fees
+                coin.inflation = inflation
+                coin.hashrate = hashrate
+                coin.difficulty = difficulty
+                coin.stocktoflow = stocktoflow
+                coin.priceusd = priceusd
+                coin.pricebtc = pricebtc
+                coin.supply = supply
+                coin.save()
+            except:
+                pass
 
     ###################################################################################
-    dt = datetime.datetime.now(timezone.utc).timestamp()
-    symbol = 'xmr'
 
-    try:
+    supply = locale.format('%.0f', supply, grouping=True)
-        rank = list(Rank.objects.order_by('-date'))[0]
+    inflation = locale.format('%.2f', inflation, grouping=True)+'%'    
-    except:
-        message = 'Page under maintenance. Check back in a few minutes.'
-        context = {'message': message}
-        return render(request, 'charts/maintenance.html', context)
-
-    if rank.date < date.today():
-        data = update_rank()    
-        update_dominance(data)
-
-    coin = list(Coin.objects.filter(name=symbol).order_by('-date'))[0]
-    if coin:
-        now_inflation = coin.inflation
-        supply = int(coin.supply)*10**12
-        now_units = supply/(10**12)
-    else:
-        message = 'Website under maintenance. Check back in a few minutes'
-        context = {'message': message}
-        return render(request, 'charts/maintenance.html', context)
-
-    now_units = locale.format('%.0f', now_units, grouping=True)
-    now_inflation = locale.format('%.2f', now_inflation, grouping=True)+'%'
-
-    
     dt = 'index.html ' + locale.format('%.2f', datetime.datetime.now(timezone.utc).timestamp() - dt, grouping=True)+' seconds'
     print(dt)
-    context = {'now_inflation': now_inflation, 'now_units': now_units}
+    context = {'inflation': inflation, 'supply': supply}
     return render(request, 'charts/index.html', context)
 
 def pt(request):

requirements.txt

@@ -1,21 +1,50 @@
+aiohttp==3.8.1
+aiosignal==1.2.0
 asgiref==3.3.1
+async-timeout==4.0.2
+attrs==22.1.0
 beautifulsoup4==4.9.3
+cachetools==5.2.0
 certifi==2020.12.5
 chardet==4.0.0
+charset-normalizer==2.1.1
+click==8.1.3
 dj-database-url==0.5.0
 dj-static==0.0.6
 Django==3.1.7
 django-bootstrap4==2.3.1
+feedparser==6.0.10
+frozenlist==1.3.1
+google-api-core==2.8.1
+google-api-python-client==2.49.0
+google-auth==2.6.6
+google-auth-httplib2==0.1.0
+google-auth-oauthlib==0.5.1
+googleapis-common-protos==1.56.2
 gunicorn==20.0.4
+httplib2==0.20.4
 idna==2.10
+multidict==6.0.2
 numpy==1.20.1
+oauthlib==3.2.0
 pandas==1.2.3
+protobuf==3.20.1
+psaw==0.1.0
 psycopg2-binary==2.8.6
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pygsheets==2.0.5
+pyparsing==3.0.9
 python-dateutil==2.8.1
 pytz==2021.1
 requests==2.25.1
+requests-oauthlib==1.3.1
+rsa==4.8
+sgmllib3k==1.0.0
 six==1.15.0
 soupsieve==2.2
 sqlparse==0.4.1
 static3==0.7.0
+uritemplate==4.1.1
 urllib3==1.26.3
+yarl==1.8.1