Opened 2 years ago

Last modified 7 weeks ago

#898 new defect

WebStat: wrong date handling on certain days

Reported by: jcaffaro Owned by:
Priority: minor Milestone:
Component: WebStat Version:
Keywords: Cc:

Description

The following exception is currently produced when trying to view statistics (key statistics and custom events) for "the last 3 months" with format "flot" (due to February having only 28th days? This one 29th? Or every month with less than 31 days?):

* 2012-02-07 09:50:36 -> ValueError: day is out of range for month (webstat_engine.py:1922:create_graph_trend_flot)

** User details
                                     agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0
                                     group: []
                                     guest: 0
                                  language: en
                              login_method: Local
                                  nickname: admin
precached_permitted_restricted_collections: []
                        precached_useadmin: True
                       precached_usealerts: True
                      precached_useapprove: True
                      precached_usebaskets: True
                       precached_usegroups: True
                        precached_useloans: True
                     precached_usemessages: True
             precached_usepaperattribution: True
                   precached_usepaperclaim: True
                        precached_usestats: True
                   precached_viewclaimlink: False
                 precached_viewsubmissions: True
                                   referer: </stats/comments_frequency?ln=en&collection=All&format=flot&timespan=last+month&s_date=02%2F07%2F2012+00%3A00&f_date=02%2F07%2F2012+09%3A50&action_gen=Generate>
                               remote_host: 
                                 remote_ip: 127.0.0.1
                                   session: 68b0ae9eeefbec1db979872c7d942cbb
                                       uid: 1
                                       uri: </stats/comments_frequency?ln=en&collection=All&format=flot&timespan=last+three+months&s_date=02%2F07%2F2012+00%3A00&f_date=02%2F07%2F2012+09%3A50&action_gen=Generate>
                   websearch_group_records: 10
                         websearch_helpbox: 1
                       websearch_latestbox: 1

** Traceback details 

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/invenio/webinterface_handler_wsgi.py", line 462, in application
    ret = invenio_handler(req)
  File "/usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py", line 360, in _profiler
    return _handler(req)
  File "/usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py", line 407, in _handler
    return root._traverse(req, path, False, guest_p)
  File "/usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py", line 237, in _traverse
    return obj._traverse(req, path, do_head, guest_p)
  File "/usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py", line 250, in _traverse
    result = _check_result(req, obj(req, form))
  File "/usr/lib/python2.6/dist-packages/invenio/webstat_webinterface.py", line 238, in comments_frequency
    body=perform_display_keyevent('comments frequency', argd, req, ln=ln),
  File "/usr/lib/python2.6/dist-packages/invenio/webstat.py", line 1214, in perform_display_keyevent
    os.path.basename(filename), settings, ln=ln) + sql
  File "/usr/lib/python2.6/dist-packages/invenio/webstat.py", line 1520, in _perform_display_event
    create_graph_trend(data, path, settings)
  File "/usr/lib/python2.6/dist-packages/invenio/webstat_engine.py", line 1687, in create_graph_trend
    create_graph_trend_flot(trend, path, settings)
  File "/usr/lib/python2.6/dist-packages/invenio/webstat_engine.py", line 1922, in create_graph_trend_flot
    next = current.replace(day=31)
ValueError: day is out of range for month

** Stack frame details

Frame application in /usr/lib/python2.6/dist-packages/invenio/webinterface_handler_wsgi.py at line 479
-------------------------------------------------------------------------------
       476             else:
       477                 req.flush()
       478         except:
---->  479             register_exception(req=req, alert_admin=True)
       480             if not req.response_sent_p:
       481                 req.status = HTTP_INTERNAL_SERVER_ERROR
       482                 req.headers_out['content-type'] = 'text/html'
-------------------------------------------------------------------------------
	      start_response =  '<built-in method start_response of mod_wsgi.Adapter object at 0xb76819f8>'
	             environ =  [...]
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'
	    possible_handler =  'None'
	     possible_module =  'None'

Frame _profiler in /usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py at line 360
-------------------------------------------------------------------------------
       357                     raise Exception('Debugging requested, but no debugger registered: "%s"' % args['debug'])
       358             return _handler(req)
       359         else:
---->  360             return _handler(req)
       361 
       362     def _handler(req):
       363         """ This handler is invoked by mod_python with the apache request."""
-------------------------------------------------------------------------------
	            _handler =  '<function _handler at 0x23edeb1c>'
	                args =  "{'timespan': ['last three months'], 'ln': ['en'], 'format': ['flot'], 'f_date': ['02/07/2012 09:50'], 'collection': ['All'], 'action_gen': ['Generate'], 's_date': ['02/07/2012 00:00']}"
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'

Frame _handler in /usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py at line 449
-------------------------------------------------------------------------------
       446 
       447             ## as suggested in
       448             ## <http://www.python.org/doc/2.3.5/lib/module-gc.html>
---->  449             del gc.garbage[:]
       450 
       451     return _profiler
       452 
-------------------------------------------------------------------------------
	     allowed_methods =  "('GET', 'POST', 'HEAD', 'OPTIONS')"
	                   g =  'None'
	             guest_p =  '0'
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'
	                 uri =  "'/stats/comments_frequency'"
	                path =  "['stats', 'comments_frequency']"
	                root =  '<invenio.webinterface_layout.WebInterfaceInvenio object at 0x23edf26c>'
	            bad_msie =  'None'

Frame _traverse in /usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py at line 237
-------------------------------------------------------------------------------
       234         # renderer. We even pass it the parsed form arguments.
       235         if path:
       236             if hasattr(obj, '_traverse'):
---->  237                 return obj._traverse(req, path, do_head, guest_p)
       238             else:
       239                 raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
       240 
-------------------------------------------------------------------------------
	                 obj =  '<invenio.webstat_webinterface.WebInterfaceStatsPages object at 0x23edf14c>'
	                name =  "'stats'"
	             guest_p =  '0'
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'
	           component =  "'stats'"
	             do_head =  'False'
	                path =  "['comments_frequency']"
	                self =  '<invenio.webinterface_layout.WebInterfaceInvenio object at 0x23edf26c>'

Frame _traverse in /usr/lib/python2.6/dist-packages/invenio/webinterface_handler.py at line 250
-------------------------------------------------------------------------------
       247                 req.uri not in CFG_NO_LANG_RECOGNITION_URIS:
       248             ln = get_preferred_user_language(req)
       249             form.add_field('ln', ln)
---->  250         result = _check_result(req, obj(req, form))
       251         return result
       252 
       253     def __call__(self, req, form):
-------------------------------------------------------------------------------
	                 obj =  '<bound method WebInterfaceStatsPages.comments_frequency of <invenio.webstat_webinterface.WebInterfaceStatsPages object at 0x23edf14c>>'
	                name =  "'comments_frequency'"
	                form =  "{'timespan': [Field('timespan', 'last three months')], 'ln': [Field('ln', 'en')], 'format': [Field('format', 'flot')], 'f_date': [Field('f_date', '02/07/2012 09:50')], 'collection': [Field('collection', 'All')], 'action_gen': [Field('action_gen', 'Generate')], 's_date': [Field('s_date', '02/07/2012 00:00')]}"
	             guest_p =  '0'
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'
	           component =  "'comments_frequency'"
	             do_head =  'False'
	                path =  '[]'
	                self =  '<invenio.webstat_webinterface.WebInterfaceStatsPages object at 0x23edf14c>'

Frame comments_frequency in /usr/lib/python2.6/dist-packages/invenio/webstat_webinterface.py at line 238
-------------------------------------------------------------------------------
       235                 ln=ln)
       236 
       237         return page(title="Comments frequency",
---->  238                     body=perform_display_keyevent('comments frequency', argd, req, ln=ln),
       239                     navtrail="""<a class="navtrail" href="%s/stats/%s">Statistics</a>""" % \
       240                     (CFG_SITE_URL, (ln != CFG_SITE_LANG and '?ln=' + ln) or ''),
       241                     description="CDS, Statistics, Comments frequency",
-------------------------------------------------------------------------------
	            auth_msg =  "'Authorization granted'"
	                form =  "{'timespan': [Field('timespan', 'last three months')], 'ln': [Field('ln', 'en')], 'format': [Field('format', 'flot')], 'f_date': [Field('f_date', '02/07/2012 09:50')], 'collection': [Field('collection', 'All')], 'action_gen': [Field('action_gen', 'Generate')], 's_date': [Field('s_date', '02/07/2012 00:00')]}"
	                  ln =  "'en'"
	                self =  '<invenio.webstat_webinterface.WebInterfaceStatsPages object at 0x23edf14c>'
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'
	           user_info =  "{'uid': 1, 'agent': 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0', 'uri': '/stats/comments_frequency?ln=en&collection=All&format=flot&timespan=last+three+months&s_date=02%2F07%2F2012+00%3A00&f_date=02%2F07%2F2012+09%3A50&action_gen=Generate', 'precached_usegroups': True, 'session': '68b0ae9eeefbec1db979872c7d942cbb', 'precached_useadmin': True, 'group': [], 'login_method': 'Local', 'websearch_latestbox': 1, 'precached_usepaperattribution': True, 'precached_usepape [...]
	           auth_code =  '0'
	                argd =  "{'f_date': '02/07/2012 09:50', 'ln': 'en', 'format': 'flot', 'timespan': 'last three months', 'collection': 'All', 's_date': '02/07/2012 00:00', 'sql': 0}"

Frame perform_display_keyevent in /usr/lib/python2.6/dist-packages/invenio/webstat.py at line 1214
-------------------------------------------------------------------------------
      1211     else:
      1212         sql = ''
      1213     return out + _perform_display_event(data,
----> 1214                         os.path.basename(filename), settings, ln=ln) + sql
      1215 
      1216 
      1217 def perform_display_customevent(ids=[], args={}, req=None, ln=CFG_SITE_LANG):
-------------------------------------------------------------------------------
	              t_args =  "{'xtic_format': '%b', 't_format': '%Y-%m-%d', 't_fullname': 'Last three months', 't_start': '2011-12-01', 'collection': 'All', 'granularity': 'month', 't_end': '2012-02-07'}"
	                args =  "{'f_date': '02/07/2012 09:50', 'ln': 'en', 'format': 'flot', 'timespan': 'last three months', 'collection': 'All', 's_date': '02/07/2012 00:00', 'sql': 0}"
	               force =  'False'
	            settings =  '{\'multiple\': None, \'format\': \'flot\', \'xtic_format\': \'%b\', \'title\': \'Comments frequency in collection "All"\', \'xlabel\': \'Last three months (month)\', \'ylabel\': \'Number of comments\'}'
	                  ln =  "'en'"
	                data =  "[('2011-12-01 00:00:00', 18L), ('2012-01-01 00:00:00', 5L), ('2012-02-01 00:00:00', 0)]"
	                 req =  '<invenio.webinterface_handler_wsgi.SimulatedModPythonRequest object at 0x23edf94c>'
	                list =  'False'
	               param =  "'collection'"
	            filename =  "'webstat_comments_frequency_All_last_three_months'"
	            event_id =  "'comments frequency'"
	             options =  "{'timespan': ('combobox', 'Time span', [('today', 'Today', '2012-02-07', '2012-02-08', 'hour', '%Y-%m-%d', '%H'), ('this week', 'This week', '2012-02-06', '2012-02-08', 'day', '%Y-%m-%d', '%a'), ('last week', 'Last week', '2012-01-30', '2012-02-06', 'day', '%Y-%m-%d', '%a'), ('this month', 'This month', '2012-02-01', '2012-02-08', 'day', '%Y-%m-%d', '%d'), ('last month', 'Last month', '2012-01-01', '2012-02-01', 'day', '%Y-%m-%d', '%d'), ('last three months', 'Last three months', '2011-12-01',  [...]
	                parr =  "'The comments frequency is the amount of comments written for all the documents.'"
	                 sql =  "''"
	             choosed =  "{'timespan': 'last three months', 's_date': '02/07/2012 00:00', 'f_date': '02/07/2012 09:50', 'collection': 'All', 'format': 'flot'}"
	            gatherer =  '<function <lambda> at 0x23ee9454>'
	               order =  "['collection', 'format', 'timespan']"
	       allow_refresh =  'True'
	                 out =  '\'<p>The comments frequency is the amount of comments written for all the documents.</p><form method="get">\\n        <input type="hidden" name="ln" value="en" /><table id="keyevent_table" class="searchbox">\\n                    <thead>\\n                        <tr><th class="searchboxheader">Collection</th><th class="searchboxheader">Output format</th><th class="searchboxheader">Time span</th><th class="searchboxheader"></th></tr>\\n                </thead>\\n                <tbody><tr valig [...]

Frame _perform_display_event in /usr/lib/python2.6/dist-packages/invenio/webstat.py at line 1520
-------------------------------------------------------------------------------
      1517         create_graph_table(data, path, settings)
      1518         return TEMPLATES.tmpl_display_event_trend_text(settings["title"], path, ln=ln)
      1519 
----> 1520     create_graph_trend(data, path, settings)
      1521     if settings["format"] == "asciiart":
      1522         out = TEMPLATES.tmpl_display_event_trend_ascii(
      1523             settings["title"], path, ln=ln)
-------------------------------------------------------------------------------
	                  ln =  "'en'"
	                path =  "'/opt/cds-invenio/var/www/img/tmp_webstat_comments_frequency_All_last_three_months'"
	                data =  "[('2011-12-01 00:00:00', 18L), ('2012-01-01 00:00:00', 5L), ('2012-02-01 00:00:00', 0)]"
	                name =  "'webstat_comments_frequency_All_last_three_months'"
	            settings =  '{\'multiple\': None, \'format\': \'flot\', \'xtic_format\': \'%b\', \'title\': \'Comments frequency in collection "All"\', \'xlabel\': \'Last three months (month)\', \'ylabel\': \'Number of comments\'}'

Frame create_graph_trend in /usr/lib/python2.6/dist-packages/invenio/webstat_engine.py at line 1687
-------------------------------------------------------------------------------
      1684     elif settings["format"] == 'gnuplot':
      1685         create_graph_trend_gnu_plot(trend, path, settings)
      1686     elif settings["format"] == "flot":
----> 1687         create_graph_trend_flot(trend, path, settings)
      1688 
      1689 
      1690 def create_graph_trend_ascii_art(trend, path, settings):
-------------------------------------------------------------------------------
	               trend =  "[('2011-12-01 00:00:00', 18L), ('2012-01-01 00:00:00', 5L), ('2012-02-01 00:00:00', 0)]"
	                path =  "'/opt/cds-invenio/var/www/img/tmp_webstat_comments_frequency_All_last_three_months'"
	            settings =  '{\'multiple\': None, \'format\': \'flot\', \'xtic_format\': \'%b\', \'title\': \'Comments frequency in collection "All"\', \'xlabel\': \'Last three months (month)\', \'ylabel\': \'Number of comments\'}'

Frame create_graph_trend_flot in /usr/lib/python2.6/dist-packages/invenio/webstat_engine.py at line 1922
-------------------------------------------------------------------------------
      1919         current =  _to_datetime(maxx, '%Y-%m-%d %H:%M:%S')
      1920         next = current + granularity_td
      1921         if (granularity_td.seconds + granularity_td.days * 24 * 3600) > 2592000:
----> 1922             next = current.replace(day=31)
      1923         tics += 'xaxis: { mode:"time",min:parseDate%s("%s"),max:parseDate%s("%s")},'\
      1924             % (title, _to_datetime(minx, '%Y-%m-%d %H:%M:%S'), title, next)
      1925 
-------------------------------------------------------------------------------
	                size =  "['500', '400']"
	                minx =  "'2011-12-01 00:00:00'"
	                maxx =  "'2012-02-01 00:00:00'"
	                rows =  '3'
	               title =  "'Commentsfr'"
	               trend =  "[('2011-12-01 00:00:00', 18L), ('2012-01-01 00:00:00', 5L), ('2012-02-01 00:00:00', 0)]"
	            settings =  '{\'multiple\': None, \'format\': \'flot\', \'xtic_format\': \'%b\', \'title\': \'Comments frequency in collection "All"\', \'xlabel\': \'Last three months (month)\', \'ylabel\': \'Number of comments\'}'
	                next =  'datetime.datetime(2012, 3, 3, 0, 0)'
	             current =  'datetime.datetime(2012, 2, 1, 0, 0)'
	                tics =  "'yaxis: {\\n                tickDecimals : 0\\n        },'"
	      granularity_td =  'datetime.timedelta(31)'
	                path =  "'/opt/cds-invenio/var/www/img/tmp_webstat_comments_frequency_All_last_three_months'"
	               first =  '1'
	                 out =  '\'<!--[if IE]><script language="javascript" type="text/javascript"\\n                    src="/js/excanvas.min.js"></script><![endif]-->\\n              <script language="javascript" type="text/javascript" src="/js/jquery.flot.min.js"></script>\\n              <script language="javascript" type="text/javascript" src="/js/jquery.flot.selection.min.js"></script>\\n              <script id="source" language="javascript" type="text/j [...]
	                 row =  '2'

Change History (1)

comment:1 Changed 7 weeks ago by simko

See also ticket:1715

Note: See TracTickets for help on using tickets.