lxml

Known Gotchas

• ⚠ XPath returns list of nodes not single value — root.xpath('//item/text()') returns list ['val1', 'val2']; root.xpath('//item[1]/text()') returns list with one element; agent code: use [0] or xpath('//item[1]/text()')[0] for single value; or xpath('string(//item[1]/text())') returns string directly; xpath('count(//item)') returns float
• ⚠ lxml elements have tail text — etree structure: <a>text<b/>tail</a>; elem.text is 'text' (before child); elem[0].tail is 'tail' (after child b, still part of parent a); agent code parsing mixed-content XML: use itertext() or get_text() from lxml.html for HTML; raw XML: concatenate elem.text and each child's tail
• ⚠ Namespace handling in XPath — <root xmlns:ns='http://example.com'><ns:item/></root>; xpath('//ns:item') requires namespace map: root.xpath('//ns:item', namespaces={'ns': 'http://example.com'}); agent code with namespaced XML: always define namespace map; Clark notation alternative: root.xpath('//{http://example.com}item')
• ⚠ iterparse memory management — iterparse yields (event, elem) pairs; element still holds children in memory; agent code: after processing elem, call elem.clear() and del elem; also root reference keeps everything alive: use root = tree.getroot() outside loop or clear root too; without clearing: iterparse for large files still loads all into memory
• ⚠ from lxml import etree vs import lxml.etree — both work; from lxml import etree is conventional; lxml.html is separate: from lxml import html; html module has fromstring() that handles malformed HTML better than etree; agent code: use html.fromstring() for HTML (returns HtmlElement); etree.fromstring() for strict XML
• ⚠ html.fromstring() vs html.document_fromstring() — html.fromstring('<p>text</p>') may return an Element (not full document) if input is a fragment; html.document_fromstring() always returns full document HtmlElement; agent code parsing complete HTML pages: use document_fromstring(); for fragments: fromstring() and work with the element directly